All right then, so you want to make a opening book from scratch. A one of a kind book that is your own. A book that in fact can be proprietary. Let me see if I can start you on the right track.
Although there are many ways to go about this let me describe the method that I have found to be the most successful. It Involves a process of development over time.
The first thing you will need is a few comprehensive databases of engine games played under rated conditions and times. (it is generally accepted that we do not use human games in engine books) I recommend using the available game bases from PlayChess server. They can be found in several places on the internet. Try to get the most current ones (all 2007 games and all Jan, 2008) are available. These are quite large databases; but can just be manipulated by the Fritz 9, 10, 11 GUIs.
It will probably be to your advantage to look around on the net for the most current game bases.
I have all PlayChess engine room games from 2006, 2007 and 2008; however at this time I do not have the space on my web site to make them directly available. With out question the most comprehensive engine game bases can be acquired from Richard Stickles. (the playchess engine room Sysop) Richard seem to make the games available for download on an irregular schedule. So the best place to start may be Spaghetti Chess it appears that most of the 2007 games and Jan, 2008 games can be downloaded here… spaghettichess.com
Also I like to add the last two PAL/CSS Freestyle Chess Tournaments to the mix as new and innovative lines are often introduced here… Freestyle Chess Games
Ok, you now have large game bases of current engine games. Tens of thousands of them. All mixed together; good games, bad games, winning lines and losing ones. After combining all bases into one master base the first thing you will need to do is a bit of filtering.
Bring your master base up on the Fritz GUI and go to filter games. Lets get rid of all games with less than 35 moves. Set filter for 1-35 moves, select all, delete, remove deleted games from base. Do the same with the draws, set filter to bring up all draws then select all, delete, remove deleted games.
Now open up an empty data base and name it New Book Main Base (or something like Blitz Book Games) This will be one of two game bases that you will continue to add games to over a period of time.
Go back to the master game base and this time we will do something radical. We are going to take only the games played by the top rated players. Set filter to 2700 to 3100 Elo and check both and engage. This will return 40,000 to 45,000 games from the highest rated players. (for a smaller, but more directed base try 2750-3100 Elo) Hit select all and copy. Go to your newly created game base and paste these games there.
The next thing you will need is a date base of game to be learned. These games should come from the main game base that you just created. My method for choosing the games to be learned by the book is less than scientific but extremely practical. Pick 4 or 5 of the top players of your choice, maybe.
1. Big Mc
2. Big Rabbit
Make an empty game base and name it New book Learn (or Blitz book learn, whatever you decide to name your book) Filter the main book base for Big Mc wins as white; copy and paste to the new learn book base. Do the same for Big Mc wins as black, and so on down the list. What you are doing is taking only these top players winning games and adding the to your book learn file. This will give your fledgling book the playing characteristics of these players. (basically stealing their best lines)
You are now ready to turn these new game bases into what I call a seed book. This book will be the starting point for your personal book to be developed from. In the Fritz GUI open a new empty book and name it. Import all games from the main book base that you have created. (to the maximum depth 100 moves) Now go to Learn from Database and click learn white and black wins (do not choose learn loses) and select your new learn file. (made from the top players wins)
You now have a seed book.
Go on line and play the book using these book settings.
Book settings: (Seed Book)
Use Book: on
Tournament Book: on
Varity of Play: max – (all the way to left)
Influence of learn Value: max + (all the way to right)
Learning Strength: max – (all the way to left)
Minimum games: 0
Up to move: 100
Save all games that you play with the new book as these are the games that, played with your own hardware will be instrumental in its development.
In the next post I will talk about how to add the games you play with the book in a regular way as to improve and expand its capabilities.
As we separate the game of chess down into three parts (The opening, the middle game, and the end game) so too do I divide Engine Chess into three components. My thinking is, that to have a strong playing system, that is capable of sustaining a uniformly high Elo rating, detailed attention should be given to each of these aspects of engine chess.
1. Hardware: CPUs, RAM, Hard disk drives, Motherboards.
2. Software: UCI Engines, GUIs, Databases.
3. Books and EGTBs: Opening books, End game table bases.
The relative importance of these aspects seems to be in constant flux. (so I have not listed them here in order of weight) It also seems to be a highly debated point as engine players approach the game from different points of view. I am certain that players come from several different areas of expertise. Most notably Computers, Programming, and Chess. Some players enjoy seeing how there powerful computer hardware fairs in competition with other machines. Some like to use many different programs, tweaking them and in some cases fiddling with the code. Other come from the world of chess and are enticed by the strangely beautiful games produced by engine play. This I believe is the fulcrum that the fledgling sport can utilize to grow, its attraction to several different groups of enthusiasts.
As they once put on old maps beyond the explored areas “past this point there be monsters” Have no illusions that your old desktop PC with a Pentium in it will be competitive in online engine play. (that’s not to say you can’t still have fun) However there are some real monsters out there. We are now in the era of 64bit multiple core CPUs. The current median standard at this time is Intel’s Quad core 64bit chips. Search depths with these processors even at blitz time control often exceed 21 half moves. Over clocking is common and is a dark art in its self. Some 16-32 core machines lurk in the shadows ready to eat your lunch or Elo as the case my be.
Engine hash size is no longer as dependent upon RAM as it once was. The modern chess engines make use of CPUs L2 catch. (which is much larger in the new CPUs) O no, maybe I should not have said that. (this seems to be one of Rybka little secrets) The L2 catch has traditionally been used for video processing the reason for this is its much faster for the program to save temporary data and retrieve it. The L2 runs at the full speed of the CPU and the RAM modules on the motherboard only a fraction of that speed.
In fact it may be that the latest release of Rybka 2.3.2a has its non-configurable L2 hash size set at 128Mb. How this size was arrived at and whether or not it is the optimum setting seems to be preparatory knowledge. (If this is just idle speculation on my part perhaps Vas will add a comment and clarify the issue)
As for Hard Disk Drives; What I do is keep one just for engine play online. The only things I put on it are a stripped down windows operating system, (I like XPpro 64) The GUI (I like Fritz 9 with latest update) a few UCI engines and all the EGTBs that I will be using in play. (I can just get it all on a 150Gb 10,000rpm Raptor) A fast HD does seem to aide EGTB accesses speed.
The chess engine has reached a state of development as to be unfathomable to us ordinary mortals. Rybka 2.3.2a is by far the strongest commercially available engine at this time. There are some contenders especially in long time controls. Zappa Chess engine by Anthony Cozzie recently beat Rybka in an exhibition match in Mexico It is believed that the Zappa Mexico program used was better at its usage of more than 4 core CPUs. (This is alluding to the perceived problem that Rybka has in correct scaling above 4 cores)
Most of the GUIs are vary good; Fritz 9-10-11 are all compatible with UCI engine as are Shredder and Hiarcs.
ChessBase 9.0 is the premier database program available. Although some what expensive it is worth its price to the serious game collector. Its ability to manipulate game bases is awesome.
I feel that it is important to mention here that one should always buy these programs from the copyright holder. Not only does profiting form ones intellectual property act as an incentive for further improvement in the case of the engines you will want the authors to send you periodic updates.
Books and EGTBs
For me the opening book is the heart and soul of this kind of chess. I will be writing more in the weeks to come on my techniques and suggestions on how to make and develop books for chess engines. For right now let me just say that every one should try to make their own books. It gives your engine games a distinct caricature and unique stile that is a reflection of you own opening theories. There is much satisfaction to be had when your lines work out and much work to be done when they do not.
All the commercially available chess engines come with fairly comprehensive opening books. Although in most cases they are to broad and to shallow in scope. (The books that come with Rybka and Fritz seem to me to be intended to play against human opponents) It is relatively easy to tweak these already wide-ranging books into a more focused repertoire. This probably is where most players should start in the quest to take control of the stile that their engines will play. (Much more about opening books in later entries)
EGTBs are simply the game of chess worked out to its conclusion when only a few peaces are left on the board. All 3-4-5 and most 6 man bases are available some where on the net. However you must be aware that having all 3-4-5-6 man egtbs will require you to have at least 1.4Tb of storage space. No mater how fast your Hard Drives are this will prove to be to taxing on your system in fast time controls.
I recommend getting ChessBase Endgame Turbo 3 Nalimov Tablebases it comes on 9 DVDs and has all 3-4-5 man and a few 6 man. Once loaded onto your HD it will be about 42Gb in size. If configured properly in your GUI it will give you about +30 Elo in engine play. A list of the most commonly accruing 6 man end games can be found on the internet. If you feel brave and have the time and space you can use this list to download a further 100Gb of egtbs from the net. (This will take some time) 140-150Gb of the right bases can get you a +70 Elo boost in fast time controls.