In the last few months a new website has made it appearance upon the net. One that’s mission statement and execution is so aligned with my viewpoint of what is best for the advancement of computer chess that I must shout its praises.
Norm Pruitt’s: www.enginechess.com
The mission statement of Norm’s site is to attempt to bring the computer chess player the tools needed to be competitive in a computer vs. computer environment. In this he is succeeding superbly, where others have fallen short. His site is clear, easy to navigate and very comprehensive.
Here you will find opening books by some of the top engine players (many found nowhere else). Articles about and interviews with Freestyle players. Links to all the right websites to get the information that is needed to increase your Elo in engine play. A new forum has just been established there and I will be closely monitoring it and adding to the discussions.
If you need to down load EGTB this is the place to go.
Norm is an acknowledged expert on the use of end game table bases and the hardware require to run them as well as a innovative book maker. He has held the top ranking in engine play, both in Blitz and longer time controls many times over the years. He resides in North Carolina with his wife Joyce and Sissy the chess cat.
I have been receiving a considerable amount of correspondence relating to my book making theories. The overwhelming majority of it positive in nature. Thank you all for your kind words. I have started posting on my website some of the more representative comments and questions. Look here Book Making Q & A Please do not hesitate to email me with your inquires, I will try to respond to all. Kevin@frayerchess.com
The most ask question is; can I have the Password to download the books for book testers. (the most current updates) The answer unfortunately is no. These are the books that the book testing team that works with me are currently engaged in playing online. It is from these games played under rated conditions that the updates are derived. In order to give the dedicated testing team a competitive advantage I do not release an update to the public until I am sure that it has been superseded by the latest update available to the team.
Never the less I feel that the public updates are very strong and used correctly by the serious player will provide a clear advantage, especially in blitz play.
One more quick word about this new chess blog. The article you just read was originally posted on my blogspot.com Blog, February 21,2008. I am transferring all my older post from blogspot.com over to WorldPress.com. I will soon be caught up and hope to maintain both blogs. I have been advised that this new format on WorldPress is easer to read and will reach a more diverse audience.
Ok, you have a tournament book that is focused and plays a narrow variety of opening moves to a good depth. Now you need to hand tune it and keep it current. But what exactly are you tuning it to? The answer is; primarily your entire playing system. Which includes your hardware and its capabilities, your engine and its configuration settings, as well as the ever changing lines being played by your opponents.
There is no such thing as a perfect book that has all the answers to all the lines. A good opening book is a thing that is in constant flux. While in the human chess world an opening repertoire may serve a player for many years with only minor changes, it is not so in engine chess. Computers have a way of finding their way around seemingly impassable winning lines with frightening elegances and speed.
To get started; go on line and play 50-60 games at your desired time control. A hand tuned book is most certainly sensitive to time control changes. (by that I mean use one book for 3-0 play one for 16-0 and another for 60-0 + times) You may use the same book but with different tuning. Do not fiddle around with your engine configuration settings to much. (If you are using Rybka 2.3.2a the default settings are the strongest)
Now take the games that you have just played and go off line and into the engine GUI. Open up the My Internet Machine Games database file and do a bit of weeding. At first just go ahead and delete all the draws and games that go fewer than 35-40 moves. Now physically look at the games that you won. Games that you won from opponents that had inferior hardware to yours can be deleted. This should leave you with the games that you lost and a few that you won.
Take these games and get to work, the idea is to alter the opening book manually not only to avoid the loses but to find a winning line form before the point your evaluation number went negative. Make sure that you have the opening book that you wish to alter loaded and go to the book display. Right click the mouse on the book display window and activate allow move adding. Now any moves that you make manually while the engine is in infinite analysis mode will be added to the book. (as green play in tour moves)
If the book line went wrong before the engine engaged it may be necessary to mark some book moves red and try a deferent way. (book moves are turn red by right clicking on the moves and choosing the do not play in tournament option)
When the engine analysis mode is used (in most engines) it reverts to a brute force search. Where as in normal play all kinds of pruning tricks are used to reduce the number of moves the engine must look at. My point is that when going over played games in analyses mode you can often find strong moves that Rybka discarded in early pruning as being unproductive. It take some time and patients to get the search depth to an effective level. (time dependent on your hardware of course) When you are analyzing a game that you have lost, that had an average search depth of 16 ply you will probably need to go 19-20 ply in a brute force search starting several moves before the detectable swing in positional value to find a good refuting line.(more incomprehensible explanations on the dark art of book tuning early in the next week)
Although the opening book only has control of the first 50 moves. (100 half moves) they are most important of the game. There is no reason why (given a bit of practice) that you can not prevent your engine from losing in the first half of the game. No mater what hardware you have it is not difficult to duplicate the depth of the fastest CPUs in offline analyses. (it just takes a bit more time)
When reviewing your games played online give the most attention of course to the games that you lost, but do not forget to look at the draws and wins as well. Many times using infinite analyses it is possible to find a line that breaks the draw. In games that you have won against strong opponents lengthen the book lines. To do this use the manual add move function in the book window of the GUI. This method of adding moves to the book gives the best control but do not forget that it adds all moves as green play moves. So be sure to go back and remove the green of the move in the losing side of the line. (this is done by right clicking on the move and hitting main move a second time)
Spend more time on finding winning lines as white, as black you may have to be satisfied with getting the draw in some lines. Here are the facts about the percentages of White vs. Black wins.
W 50% – B 50% In human club play:
W 52% – B 48% In master play:
W 56% – B 44% In GM play
W 62% – B 38% In engine play
I think this data says something about the relative strength of computers over humans. But my point is that in engine chess, the opening impetus is more pronounced for white. As black you should not expect to come out of opening book with a positional advantage very often. -.30 -.50 going into the middle game as Black will still give you a fighting chance.
Trying to tell someone how to update and tune books is some what a like trying to tell someone how to type on a keyboard. You can tell them where to put the fingers and what some of the edit keys do, but the only way one learns is by doing. Along the way many mistakes will be made. (always keep a archived copy of the books that you are tuning, so if something goes very wrong you can start over) And believe me when I say that the book learning function will not produce the same result as the basic methods that I have talked about, nor will it give the same satisfaction and connection to engine chess. Roll up your sleeves and get in the opening book trenches and remember in engine chess the book is the key…
In the next few days I am going to go over my method of developing original books from databases. But I have a lot of time on my hands and if any one has a specific question about tuning please feel free to send it to me in an email and I will try to answer it. Kevin@frayerchess.com
Before I go any farther let me address this issue of stealing lines. (I used this term in the last post) It is a fact that some players out there seem to think that if they personally worked out a line or a refutation of one; that they somehow own this series of moves. I find this thinking hilariously funny. It brings to my mind Bobby Fischer jumping up from the table and saying you can’t play that move it is mine, you stole it from my game in the interzonals last year… I call this concept of stealing lines. Fischer thinking. “Look Bobby, once you play the line in open play it belongs to all of us.” This is how, in esoteric disciplines, we collectively learn and advance human knowledge. It is by the possess of peer review that the lines you submit in open play are accepted or rejected. So there is no real stealing of anything only finding and using good ideas. We are all standing on the shoulders of geniuses to be where we are.
Now that you have made your seed book and played some games with it, you are ready to start its development by adding games. Adding games to a book (both your own and kibitz ones) is much different than hand tuning. I think it would be accurate to say that adding games increases the width of the book where as, tuning deals with depth and direction.
It is always best to add games to the book that are played by the book that you are adding them to. This technique has a tendency to reinforce and expand winning lines. However if you find good opening lines while kibitzing do not hesitate to include them into your book.
Lets look at the method of adding the games. First never add games directly to an existing book. Add the games to the database that the book was made from and then make a fresh book from the updated game bases. Importing games into an existing book is probably the most conmen mistake that is made in book making.
Make sure that if you add a game to the learn game base you also add it the main game base.
I recommend that you add all games that you won in online play to the main game base and a selected few that beat you solidly within the first 50 moves. Be more selective as to the games you add to the learn base. Look carefully at your played games; making sure that advantage in the game came from the opening or early in the middle game. (Never add draws, only wins and losses)
Continue to add the games that you play with the seed book back into the book using this technique. Every day review your games, update the game bases (both main base and learn) and remake the book for the next time at online play. This process will slowly but surly expand the books practical repertoire and adjust it to your particular system engine combination. You will begin to notice games that your engine struggled to win now being won easily and good winning lines that were used against you now being played by your book.
At least 2000 games or so should be added to the seed book before you start the hand tuning process. Once you start hand tuning you can no longer add games as red-green marks would be lost.
As always; I am sure that there are many impotent details I have inadvertently forgot to mention. If you get stuck or have a problem please do not hesitate to ask me specific question via email. Kevin@frayerchess.com
All right then, so you want to make a opening book from scratch. A one of a kind book that is your own. A book that in fact can be proprietary. Let me see if I can start you on the right track.
Although there are many ways to go about this let me describe the method that I have found to be the most successful. It Involves a process of development over time.
The first thing you will need is a few comprehensive databases of engine games played under rated conditions and times. (it is generally accepted that we do not use human games in engine books) I recommend using the available game bases from PlayChess server. They can be found in several places on the internet. Try to get the most current ones (all 2007 games and all Jan, 2008) are available. These are quite large databases; but can just be manipulated by the Fritz 9, 10, 11 GUIs.
It will probably be to your advantage to look around on the net for the most current game bases.
I have all PlayChess engine room games from 2006, 2007 and 2008; however at this time I do not have the space on my web site to make them directly available. With out question the most comprehensive engine game bases can be acquired from Richard Stickles. (the playchess engine room Sysop) Richard seem to make the games available for download on an irregular schedule. So the best place to start may be Spaghetti Chess it appears that most of the 2007 games and Jan, 2008 games can be downloaded here… spaghettichess.com
Also I like to add the last two PAL/CSS Freestyle Chess Tournaments to the mix as new and innovative lines are often introduced here… Freestyle Chess Games
Ok, you now have large game bases of current engine games. Tens of thousands of them. All mixed together; good games, bad games, winning lines and losing ones. After combining all bases into one master base the first thing you will need to do is a bit of filtering.
Bring your master base up on the Fritz GUI and go to filter games. Lets get rid of all games with less than 35 moves. Set filter for 1-35 moves, select all, delete, remove deleted games from base. Do the same with the draws, set filter to bring up all draws then select all, delete, remove deleted games.
Now open up an empty data base and name it New Book Main Base (or something like Blitz Book Games) This will be one of two game bases that you will continue to add games to over a period of time.
Go back to the master game base and this time we will do something radical. We are going to take only the games played by the top rated players. Set filter to 2700 to 3100 Elo and check both and engage. This will return 40,000 to 45,000 games from the highest rated players. (for a smaller, but more directed base try 2750-3100 Elo) Hit select all and copy. Go to your newly created game base and paste these games there.
The next thing you will need is a date base of game to be learned. These games should come from the main game base that you just created. My method for choosing the games to be learned by the book is less than scientific but extremely practical. Pick 4 or 5 of the top players of your choice, maybe.
1. Big Mc
2. Big Rabbit
Make an empty game base and name it New book Learn (or Blitz book learn, whatever you decide to name your book) Filter the main book base for Big Mc wins as white; copy and paste to the new learn book base. Do the same for Big Mc wins as black, and so on down the list. What you are doing is taking only these top players winning games and adding the to your book learn file. This will give your fledgling book the playing characteristics of these players. (basically stealing their best lines)
You are now ready to turn these new game bases into what I call a seed book. This book will be the starting point for your personal book to be developed from. In the Fritz GUI open a new empty book and name it. Import all games from the main book base that you have created. (to the maximum depth 100 moves) Now go to Learn from Database and click learn white and black wins (do not choose learn loses) and select your new learn file. (made from the top players wins)
You now have a seed book.
Go on line and play the book using these book settings.
Book settings: (Seed Book)
Use Book: on
Tournament Book: on
Varity of Play: max – (all the way to left)
Influence of learn Value: max + (all the way to right)
Learning Strength: max – (all the way to left)
Minimum games: 0
Up to move: 100
Save all games that you play with the new book as these are the games that, played with your own hardware will be instrumental in its development.
In the next post I will talk about how to add the games you play with the book in a regular way as to improve and expand its capabilities.
As we separate the game of chess down into three parts (The opening, the middle game, and the end game) so too do I divide Engine Chess into three components. My thinking is, that to have a strong playing system, that is capable of sustaining a uniformly high Elo rating, detailed attention should be given to each of these aspects of engine chess.
1. Hardware: CPUs, RAM, Hard disk drives, Motherboards.
2. Software: UCI Engines, GUIs, Databases.
3. Books and EGTBs: Opening books, End game table bases.
The relative importance of these aspects seems to be in constant flux. (so I have not listed them here in order of weight) It also seems to be a highly debated point as engine players approach the game from different points of view. I am certain that players come from several different areas of expertise. Most notably Computers, Programming, and Chess. Some players enjoy seeing how there powerful computer hardware fairs in competition with other machines. Some like to use many different programs, tweaking them and in some cases fiddling with the code. Other come from the world of chess and are enticed by the strangely beautiful games produced by engine play. This I believe is the fulcrum that the fledgling sport can utilize to grow, its attraction to several different groups of enthusiasts.
As they once put on old maps beyond the explored areas “past this point there be monsters” Have no illusions that your old desktop PC with a Pentium in it will be competitive in online engine play. (that’s not to say you can’t still have fun) However there are some real monsters out there. We are now in the era of 64bit multiple core CPUs. The current median standard at this time is Intel’s Quad core 64bit chips. Search depths with these processors even at blitz time control often exceed 21 half moves. Over clocking is common and is a dark art in its self. Some 16-32 core machines lurk in the shadows ready to eat your lunch or Elo as the case my be.
Engine hash size is no longer as dependent upon RAM as it once was. The modern chess engines make use of CPUs L2 catch. (which is much larger in the new CPUs) O no, maybe I should not have said that. (this seems to be one of Rybka little secrets) The L2 catch has traditionally been used for video processing the reason for this is its much faster for the program to save temporary data and retrieve it. The L2 runs at the full speed of the CPU and the RAM modules on the motherboard only a fraction of that speed.
In fact it may be that the latest release of Rybka 2.3.2a has its non-configurable L2 hash size set at 128Mb. How this size was arrived at and whether or not it is the optimum setting seems to be preparatory knowledge. (If this is just idle speculation on my part perhaps Vas will add a comment and clarify the issue)
As for Hard Disk Drives; What I do is keep one just for engine play online. The only things I put on it are a stripped down windows operating system, (I like XPpro 64) The GUI (I like Fritz 9 with latest update) a few UCI engines and all the EGTBs that I will be using in play. (I can just get it all on a 150Gb 10,000rpm Raptor) A fast HD does seem to aide EGTB accesses speed.
The chess engine has reached a state of development as to be unfathomable to us ordinary mortals. Rybka 2.3.2a is by far the strongest commercially available engine at this time. There are some contenders especially in long time controls. Zappa Chess engine by Anthony Cozzie recently beat Rybka in an exhibition match in Mexico It is believed that the Zappa Mexico program used was better at its usage of more than 4 core CPUs. (This is alluding to the perceived problem that Rybka has in correct scaling above 4 cores)
Most of the GUIs are vary good; Fritz 9-10-11 are all compatible with UCI engine as are Shredder and Hiarcs.
ChessBase 9.0 is the premier database program available. Although some what expensive it is worth its price to the serious game collector. Its ability to manipulate game bases is awesome.
I feel that it is important to mention here that one should always buy these programs from the copyright holder. Not only does profiting form ones intellectual property act as an incentive for further improvement in the case of the engines you will want the authors to send you periodic updates.
Books and EGTBs
For me the opening book is the heart and soul of this kind of chess. I will be writing more in the weeks to come on my techniques and suggestions on how to make and develop books for chess engines. For right now let me just say that every one should try to make their own books. It gives your engine games a distinct caricature and unique stile that is a reflection of you own opening theories. There is much satisfaction to be had when your lines work out and much work to be done when they do not.
All the commercially available chess engines come with fairly comprehensive opening books. Although in most cases they are to broad and to shallow in scope. (The books that come with Rybka and Fritz seem to me to be intended to play against human opponents) It is relatively easy to tweak these already wide-ranging books into a more focused repertoire. This probably is where most players should start in the quest to take control of the stile that their engines will play. (Much more about opening books in later entries)
EGTBs are simply the game of chess worked out to its conclusion when only a few peaces are left on the board. All 3-4-5 and most 6 man bases are available some where on the net. However you must be aware that having all 3-4-5-6 man egtbs will require you to have at least 1.4Tb of storage space. No mater how fast your Hard Drives are this will prove to be to taxing on your system in fast time controls.
I recommend getting ChessBase Endgame Turbo 3 Nalimov Tablebases it comes on 9 DVDs and has all 3-4-5 man and a few 6 man. Once loaded onto your HD it will be about 42Gb in size. If configured properly in your GUI it will give you about +30 Elo in engine play. A list of the most commonly accruing 6 man end games can be found on the internet. If you feel brave and have the time and space you can use this list to download a further 100Gb of egtbs from the net. (This will take some time) 140-150Gb of the right bases can get you a +70 Elo boost in fast time controls.
Just before the turn of the new century I began to concern myself exclusively with Engine Chess. By the term, (engine chess) I mean one chess program playing another chess program. The winboard GUI had facilitated this but it was not until the Arena GUI was made available to the public in 2001 that we had a graphic interface specifically designed to play one chess engine against another.
Arena not only was compatible with the winboard protocols but also the newer UCI engines. Everything was configurable; the time controls, the number of games in a match, the engine parameters and of course the opening books along with their weighted learning capabilities. One could now take their computer hardware, chess engines and opining books onto the internet and play against other computers and their human handlers.
Commercial chess programs, particularly Chessbase’s Fritz 7-8 came along with strong client based engines and very pleasing graphics. In September of 2001 Chessbase’s player site, Playchess.com opened a room specifically for engine play and competitive engine chess came into existences. An Elo rating system based on individual computers, programs and the way their users configured them now was possible rather than be based on how they did against human Masters, This made judgments of relative engine strength more viable and contributed to programming development.
Now I know what some are thinking that there had been WCCC tournaments held every few years since 1974. But in fact these events were and still are primarily for showcasing commercial programs and cutting edge computer hardware. In my world view, these elite events held in exotic locations do not constitute Engine Chess being played in public venues by ordinary individuals that we enjoy today.
The commercial programs quickly proved themselves to be as strong their advertising departments had clamed. Fritz, Shredder, Junior, Hiarcs and many others all proved that they could play hauntingly beautiful chess. In early 2006 Vasik Rajlich’s UCI engine Rybka (little fish) rose to the top and is currently the dominant chess program in the world. One of the things I realized early about engine chess was that all things be equal, (both players having similar hardware and using the strongest program available) it is the opening book that made the difference.
Not only does the opening book set the tone of the game but its manipulation is best way for the human operator to communicate with his program. Even if an engine has detailed configuration setting, (material value of the pieces, king safety, aggressive vs. defensive stile) these are esoteric and communicate with the program in a none human way. Only the engine book guided by a human intelligence can channel a chess engines positional decision making power to its fullest potential.
It is possible to move a huge mound of dirt by hand. Using shovels, wheelbarrows and human muscle. History is replete with such accomplishments. Today however we attack the mound with bulldozers and trucks making short work of it. It is no disrespect to the laborers that would have done this feat in the past. As we move forward in 21st century applying these computer programs, theses engines of the mind to our beloved game of chess we mean not to diminish the accomplishments of the great human players but only to advance the game , the art that is chess.
Sometime in the early 90s I became aware of Dr. Robert Hyatt’s open source Crafty program. Not being a programmer, my ability to modify and manipulate the engine was limited. But I saw the potential of chess engines being separate from the graphical interface. They could be modified like internal combustion engines and then dropped into a GUI as if it were a racing chassis. This may have been common in computer programming at that time, I do not know, (compartmentalizing programs so as to be updateable) however the ability to drop an engine into the chess program was and still is a cool idea. Undoubtedly it has contributed to the development of the number crunching magic that goes on inside these invisible contraptions.
There were many good early engines that ran in the winboard GUI. (Delfi, Gromit, Little Goliath, Yace) to name but a few. Remember good is a vary relative term as I have used it here. Like Crafty they all had multiple configuration settings and could be tweaked nicely to the hardware that you were running them on. Opening books that could easily be modified came into being along with book learning and positional learning. Nalimov 3-4 man end game table bases where mated to many of these engines. Winboard protocols were superseded by UCI and we came into the modern era of computer chess.
By the end of the century the 32bit CPU had picked up speed and I will be darned if I could beat the things any longer. If the truth be known, no one could. We all had chess masters setting on our desks.
This may be an obscure reference but I believe to be an accurate analogy. In the movie Close Encounters of the Third Kind remember how after the mother ship had landed and the humans were beginning to communicate with it using musical notes and harmonics. At a given point the communication got to fast and complex for the human operators. A bank of computers could be seen in the background and a voice says “ok, we are taking over this conversation now” and the computers where turned on. The exchange continued faster and more complex than it had been. Strange yet somehow familiar as nonhuman intelligence spoke to us in a human way. This I think is what we are experiencing when we match one computer engine against a another. A strangely beautiful and somewhat disturbing display of nonhuman intelligent.
For me the beginning was in 1983 and the release of the Atari 800XL home computer. At the time I was working at Vincennes University and had had access to a main frame computer with teletype stile terminals. It had no chess programs and the Internet was just a baby. Connecting only a few universities and libraries. The Atari 800XL however had a monitor, floppy disk drive, and loadable programs that included. A word processor, a database and at least two available chess programs. (Sargon, Chessmaster)
At the time I was still playing in over the board chess tournaments and was the faculty sponsor of the universities chess club. As I began to play more correspondences chess I saw the potential of keeping these games organized in the meager database capabilities of these early chess programs. The chess engines themselves where not yet a factor in correspondents chess as they lacked the strength to be of any help in positional analyses.
I suppose, I did not foresee in those early days that programs using minimax algorithms (decision making equations in a zero sum games) would ever be strong enough to challenge the best human players. I believed the way to go was vast databases of games that would be searched for the winning move in any given position. In the early 80s CPU power and data storage were expanding exponentially.
Even on the Atari 800XLs 64K floppy disk many hundreds of games could be stored. I spent hours entering GM games mostly from the Chess Informant, that I subscribed to. These key GM games in the lines that I played in correspondence chess were invaluable in getting my postal (as we called it back then) ELO above 2100 and keep it there for many years.
I smile now when I think of those days. My entire collection of games where no more than 900Kb in size. Today in modern engine vs. engine play I use opening books that are 200-300Mb and end game table bases that are 150Gb in size. It is still not enough, perhaps it never will be.