Author: Ian

 

Author(s): Cosmo Bobak (GBR), Conor Anstey (GBR), Archishmaan Peyyety (USA)
Release Date: 2025-01-27
Language(s): Rust
Repo Owner: cosmobobak
Repo URL: https://github.com/cosmobobak/viridithas

This release brings tuned search, a new net, and some generally improved internal design.

Upgrades, in chronological order:

  • Remove material corrhist. #208
  • Separate sum vector for reproducible bench. #210
  • Guard QS SEE pruning a bit more. #211
  • Remove granular depth. #212
  • Remove is_winning from movesloop. #213
  • Use mmap to share weights between multiple processes of Viridithas. #214
  • Improve NNUE inference. #216
  • Major refactor of internals. #217
  • Two SPSA tunes. #220
  • Take advantage of bucket layout to merge king planes. #221
  • Simplify continuation history indexing. #222
  • New network, ID perseverance. #223

https://github.com/cosmobobak/viridithas/releases/download/v16.0.0/viridithas-16.0.0-linux-x86-64-v1
https://github.com/cosmobobak/viridithas/releases/download/v16.0.0/viridithas-16.0.0-linux-x86-64-v2
https://github.com/cosmobobak/viridithas/releases/download/v16.0.0/viridithas-16.0.0-linux-x86-64-v3
https://github.com/cosmobobak/viridithas/releases/download/v16.0.0/viridithas-16.0.0-linux-x86-64-v4
https://github.com/cosmobobak/viridithas/releases/download/v16.0.0/viridithas-16.0.0-win-x86-64-v1.exe
https://github.com/cosmobobak/viridithas/releases/download/v16.0.0/viridithas-16.0.0-win-x86-64-v2.exe
https://github.com/cosmobobak/viridithas/releases/download/v16.0.0/viridithas-16.0.0-win-x86-64-v3.exe
https://github.com/cosmobobak/viridithas/releases/download/v16.0.0/viridithas-16.0.0-win-x86-64-v4.exe
https://github.com/cosmobobak/viridithas/archive/refs/tags/v16.0.0.zip

This is the CTG that makes the rounds of the various chess forums. It is a very large opening book. About 11.5 GB. As a RAR file it’s still about 3.5 GB. It is a bad idea to try to use this in Fritz during an engine match or tournament, except maybe under very long time controls. It’s meant to be for correspondence players, and thus would be useful as a simple reference in either Fritz or ChessBase (or other software that can read a CTG). My understanding of correspondence chess is that engines are sometimes allowed, thus creating a need for an engine like CorChess. But for the most part they aren’t. And tablebases are sometimes allowed, but also, for the most part, verboten. The one thing they always allow for is an opening book. Thus the bigger the book, the more freedom you have to explore the moves the way you would with a chess engine. I realize that this is a bit of a dodgy approach to corr chess, but it’s not cheating, and it’s not disallowed.

https://www.mediafire.com/file/bl7hqnoc3mq500v/UpdateImmortal+2012a+final+v31-2.rar/file

This is the engine that I got ChatGPT to put together for me. It’s written in C, and I have absolutely no hand in the coding of it. It’s a work-in-progress. So far compilation efforts have been in vain.

Ideally other people would turn it into something, but I’ll keep having the robot do revisions.

https://github.com/ianrastall/bear

Regular expressions are not limited to software or to text editors. They also have use in office apps, search forms, etc. But this is just a quick heads-up on how to run a regular expression search and replace in Notepad++, which is a Windows text editor.

Say you have a file with only one PGN file total. We can experiment with this.

[Event "Patricia - 4ku"]
[Site "Chess Nerd"]
[Date "2024.12.02"]
[Round "1"]
[White "Patricia 3.1"]
[Black "4ku 5.1"]
[Result "1-0"]
[ECO "B22"]
[GameDuration "00:05:35"]
[GameEndTime "2024-12-02T10:59:02.042 Central Standard Time"]
[GameStartTime "2024-12-02T10:53:26.553 Central Standard Time"]
[Opening "Sicilian"]
[PlyCount "129"]
[TimeControl "120+1"]
[Variation "Alapin's variation (2.c3)"]

1. e4 {+0.54/21 5.1s} c5 {-0.31/22 4.6s} 2. c3 {+0.46/21 6.2s} d5 {-0.02/22 4.8s} 3. exd5 {+0.43/24 5.7s} Qxd5 {-0.14/23 4.3s} 4. d4 {+0.38/22 6.4s} Nc6 {+0.01/23 4.1s} 5. Nf3 {+0.44/22 3.6s} Nf6 {+0.04/23 4.2s} 6. Be2 {+0.44/23 4.5s} Bf5 {-0.08/24 7.8s} 7. c4 {+1.01/22 5.7s} Qd6 {0.00/27 5.3s} 8. d5 {+1.33/22 4.0s} Nb4 {0.00/27 3.6s} 9. O-O {+1.50/21 3.8s} Nc2 {+1.32/25 4.4s} 10. Nh4 {+1.55/18 3.4s} Bg6 {+0.97/27 6.2s} 11. Nc3 {+1.63/18 3.5s} Nxa1 {+0.49/26 8.3s} 12. Nb5 {+1.85/20 3.3s} Qb8 {+0.38/27 2.6s} 13. Nxg6 {+1.90/18 3.2s} hxg6 {+1.52/25 2.5s} 14. g3 {+1.89/20 4.0s} Qc8 {+0.76/24 4.5s} 15. Re1 {+2.23/20 4.1s} Qh3 {+2.07/26 12s} 16. Nd6+ {+2.15/19 3.8s} Kd7 {+3.24/23 2.0s} 17. Qa4+ {+1.88/22 4.7s} Kxd6 {+2.50/25 4.9s} 18. Bf4+ {+1.76/23 2.6s} e5 {+0.86/26 4.2s} 19. Bxe5+ {+1.71/23 4.0s} Ke7 {+0.88/26 1.7s} 20. Bf3 {+2.57/20 2.5s} Kd8 {+0.87/26 2.1s} 21. Bxf6+ {+2.51/22 6.0s} Kc7 {+0.72/25 1.7s} 22. Be7 {+3.21/22 2.8s} Bxe7 {0.00/24 3.0s} 23. Rxe7+ {+3.83/22 3.1s} Kd6 {0.00/25 1.9s} 24. Rxb7 {+4.09/22 4.8s} Qxh2+ {0.00/26 1.6s} 25. Kf1 {+4.21/19 1.9s} Qh3+ {0.00/27 1.8s} 26. Bg2 {+3.46/20 2.3s} Qf5 {0.00/29 1.4s} 27. Qc6+ {+3.46/22 2.5s} Ke5 {-1.19/26 3.3s} 28. Re7+ {+3.60/22 1.8s} Kd4 {-1.12/28 4.6s} 29. Re4+ {+3.32/23 4.1s} Qxe4 {-1.45/25 1.2s} 30. Bxe4 {+3.45/21 1.9s} Rac8 {-1.44/25 1.3s} 31. Qd7 {+3.90/20 1.7s} Kxe4 {-1.26/24 1.8s} 32. Ke2 {+3.83/21 1.6s} g5 {-2.81/24 2.8s} 33. Qxf7 {+3.93/20 2.1s} Rce8 {-2.82/24 4.7s} 34. d6 {+4.58/20 1.7s} Ke5 {-2.65/25 1.8s} 35. d7 {+4.77/20 1.5s} Rd8 {-3.31/22 2.2s} 36. Qe7+ {+5.16/20 1.7s} Kf5 {-3.42/22 1.1s} 37. b4 {+5.22/21 2.3s} cxb4 {-2.60/22 1.1s} 38. c5 {+5.44/20 1.6s} b3 {-2.53/22 1.1s} 39. Qf7+ {+5.52/20 1.7s} Ke4 {-5.86/22 2.0s} 40. axb3 {+5.52/21 2.3s} Nxb3 {-6.84/23 0.91s} 41. Qxb3 {+5.81/23 1.9s} Kd4 {-7.01/25 1.1s} 42. c6 {+6.28/20 1.4s} Ke5 {-7.01/27 1.2s} 43. Qd1 {+6.96/20 3.5s} Ke6 {-7.01/25 0.87s} 44. Ke3 {+7.06/19 1.2s} g4 {-8.10/22 0.92s} 45. Qd4 {+7.42/19 1.6s} a5 {-9.38/25 2.2s} 46. Kd3 {+7.36/21 1.4s} a4 {-9.34/25 1.9s} 47. Kc4 {+7.44/20 1.2s} a3 {-9.75/25 1.4s} 48. Qd5+ {+8.24/18 1.6s} Ke7 {-9.85/26 1.2s} 49. Qe5+ {+9.32/21 2.4s} Kf7 {-9.87/27 0.96s} 50. Qf5+ {+9.59/21 1.3s} Ke7 {-14.81/25 1.0s} 51. Kd5 {+9.75/21 1.6s} Rxd7+ {-16.62/28 1.9s} 52. Qxd7+ {+10.93/21 1.1s} Kf6 {-299.58/30 1.7s} 53. Qe6+ {+12.14/20 1.3s} Kg5 {-28.83/23 0.78s} 54. Qe7+ {+12.18/21 2.6s} Kg6 {-299.58/25 1.5s} 55. Qxa3 {+14.15/19 1.2s} Kf7 {-299.80/25 0.78s} 56. Qa4 {+15.13/20 1.3s} Rd8+ {-299.80/24 1.2s} 57. Kc5 {+22.44/21 1.6s} Ke6 {-299.68/26 1.7s} 58. Qxg4+ {+37.58/22 1.6s} Ke5 {-299.84/27 0.84s} 59. Qg5+ {+M15/26 1.1s} Ke4 {-299.86/26 0.73s} 60. c7 {+M11/25 1.4s} Rd6 {-299.90/26 1.0s} 61. Qe3+ {+M9/25 1.3s} Kf5 {-299.92/24 0.73s} 62. c8=Q+ {+M7/26 1.6s} Kg6 {-299.94/26 0.85s} 63. Qe4+ {+M5/26 1.2s} Kg5 {-299.96/27 1.1s} 64. Qf4+ {+M3/26 1.3s} Kh5 {-299.98/26 0.96s} 65. Qcg4# {+M1/26 0.93s, White mates} 1-0

Perhaps you would like to strip out the comments. As you can see, all PGN comments are surrounded by curly brackets. These appear nowhere else in the PGN.

To use a regular expression is to look for more than simply one literal search term. In this case, you would search on  \{.*?\}

Just look at part of the file. 88. Kg1 {-299.92/34 0.69s} Kg5 … You take out the space before it, then the comment and everything inside it. When I run that on the PGN included above, I get this result:

[Event "Patricia - 4ku"]
[Site "Chess Nerd"]
[Date "2024.12.02"]
[Round "1"]
[White "Patricia 3.1"]
[Black "4ku 5.1"]
[Result "1-0"]
[ECO "B22"]
[GameDuration "00:05:35"]
[GameEndTime "2024-12-02T10:59:02.042 Central Standard Time"]
[GameStartTime "2024-12-02T10:53:26.553 Central Standard Time"]
[Opening "Sicilian"]
[PlyCount "129"]
[TimeControl "120+1"]
[Variation "Alapin's variation (2.c3)"]

1. e4 c5 2. c3 d5 3. exd5 Qxd5 4. d4 Nc6 5. Nf3 Nf6 6. Be2 Bf5 7. c4 Qd6 8. d5 Nb4 9. O-O Nc2 10. Nh4 Bg6 11. Nc3 Nxa1 12. Nb5 Qb8 13. Nxg6 hxg6 14. g3 Qc8 15. Re1 Qh3 16. Nd6+ Kd7 17. Qa4+ Kxd6 18. Bf4+ e5 19. Bxe5+ Ke7 20. Bf3 Kd8 21. Bxf6+ Kc7 22. Be7 Bxe7 23. Rxe7+ Kd6 24. Rxb7 Qxh2+ 25. Kf1 Qh3+ 26. Bg2 Qf5 27. Qc6+ Ke5 28. Re7+ Kd4 29. Re4+ Qxe4 30. Bxe4 Rac8 31. Qd7 Kxe4 32. Ke2 g5 33. Qxf7 Rce8 34. d6 Ke5 35. d7 Rd8 36. Qe7+ Kf5 37. b4 cxb4 38. c5 b3 39. Qf7+ Ke4 40. axb3 Nxb3 41. Qxb3 Kd4 42. c6 Ke5 43. Qd1 Ke6 44. Ke3 g4 45. Qd4 a5 46. Kd3 a4 47. Kc4 a3 48. Qd5+ Ke7 49. Qe5+ Kf7 50. Qf5+ Ke7 51. Kd5 Rxd7+ 52. Qxd7+ Kf6 53. Qe6+ Kg5 54. Qe7+ Kg6 55. Qxa3 Kf7 56. Qa4 Rd8+ 57. Kc5 Ke6 58. Qxg4+ Ke5 59. Qg5+ Ke4 60. c7 Rd6 61. Qe3+ Kf5 62. c8=Q+ Kg6 63. Qe4+ Kg5 64. Qf4+ Kh5 65. Qcg4# 1-0

 \{.*?\}

The space is self-explanatory. The backslash is to “escape” the left curly bracket, i.e. to keep it from being used as an actual reg exp character, instead of a search character. The next three are always together. They are a dot to say “any one character”, an asterisk to modify the dot to include as many as needed. The question mark to say, don’t get overambitious in your searching. The right curly bracket is then escaped (for the same reason the left one was) and that is really all it is.

Say you want to change the Event value to Big Tournament. You would use the following search text: ^\[Event ".*?"\]$ to indicate the beginning of a line, the tag you’re looking for, random content that you’re not keeping track of with parentheses, and then the rest of the tag and the end-of-line indicator.

^ means the beginning of a line. $ means the end of it. Backslashes escape the square brackets, and you can just replace the whole thing with \[Event "Big Tournament"\]

You don’t need to specify beginning or end of line in replace text, and since it’s always the same value, you just type it in literally.

These are two good examples to give you an idea of how to make this system work for you. It’s mostly a matter of looking things up and asking the various robots how to write the command line scripts. Things like that.

Believe it or not, with something so common, it’s actually really difficult to sort big PGN databases. ChessBase does it, but to get your PGN in there you have to convert it to CBH or 2CBH, and that takes maybe half an hour per million games. Maybe not quite that much. And then you sort, and then export to PGN. If you were doing so with a 10M game database, it would take at least a couple hours.

Scid vs PC will do it, but their limit is 5 GB for a PGN, so that cuts out half of the big projects.

Scid will do it, but only if you right click over the games list and select to Copy — Export all the games to PGN. If you go in through the regular menu system, it won’t honor the way you’ve sorted the games. It will use the original sort.

So, believe it or not, this is the only thing you can do.

That being said, if you’re ever in a total pinch, and need to sort any PGN up to maybe 5 GB without the use of anything other than a text editor, there is a way.

Notepad++ will allow you to open files up to its memory limit, and for most purposes that maxes out around 6 or 7 GB (on my system).

All you have to do is to replace the newlines inside the individual games with double tildes (~~). Those never appear in any PGN, ever. So it works.

Before you begin the long process, though, you have to make sure that your movelists are each one line. This is easy to do with pgn-extract and the -w9999 argument, or in Scid vs. PC via the regular export to PGN feature. Or you can get a script that does it. But the following process only works if every movelist is just one line long.

The main thing is to not have double tildes between games. So it has to be a bit selective. First you have to deal with the blank line between the movelist and the headers. You do so with the regular expression pair:

\]$\r\n\r\n^1.
\]~~~~1.

The first line is the search text, the second is the replace text. This says look for a right square bracket, then look to see if it’s at the end of the line, then drop two newlines, and look for a one and a dot.

The replace text says, if you find that, turn it into a right square bracket, then two tildes, then a one and a dot.

Now you can do the bigger operation. You can connect everything. You do this with:

\]$\r\n^\[
\]~~\[

As you might guess, the search text is looking for a right square bracket at the end of a line, then one newline, then a left square bracket. Keep in mind that the movelist is already taken care of. So all we’re doing is connecting the rest of the lines in every game.

Now every game is one line, with a blank line in-between. Just run a third reg exp to remove these:

\r\n\r\n
\r\n

Simple as pie. This says look for two consecutive newlines and replace them with just one. (The r stands for “carriage return” and the n stands for “line feed”. This is a newline in Windows. In Mac it’s just \r and in Linux it’s just \n. Of course that holds for all these regular expressions.)

Now if you aren’t sorting by Event (which is probably the first header) then you need to make the one you want to sort by the first header in the games. So you have to move it to the front. Let’s say you want to sort by Date.

^(.*?)\[Date "(.*?)"\]~~
~~\[Date "\2"\]~~\1

This one says to look at the beginning of every line, then remember what it saw up until it found what it was looking for. Then, when finding it, to remember what was in the quotation marks. Then to move that to the front, and put everything else behind it.

Then, with whatever text editor you’re able to do this in, you have to sort the lines up or down as you like, bringing everything into order. Now that the games are sorted, they’re all one line, and the wrong header is at the front. First, to put the header back, just repeat the previous process but instead of the text Date you would use the text Event. At that point you just replace all double tildes with newlines. Like so:

~~
\r\n

So it’s complicated, and yet it requires nothing other than a text editor.

From the old Rusbase site, which is a revision of the older Rusbase site. As has been said elsewhere (I believe) this is the best I can get, in that many of their links point to 404s. However, if you ever want to know the secret to getting these files, it’s this: if you use a link grabber, or what have you, to suck up all the URLs on the site, you’ll get a few strays, and then about 5,000 good URLs. Just change .html to .zip, put those into a download manager, and you’ll get about 3,500 ZIPs, and about 1,500 404s. Those really do point to nothing, so the system works. It’s that the site owner apparently is as neat with their files as I am, having named every ZIP and every HTML file consistently. 🙂

141,566 games (with evaluations, openings, plycount, and beauty scores):
https://www.mediafire.com/file/4snr0cu4kpyi7se/rusbase-250118.zip/file

34,005 games:
https://www.mediafire.com/file_premium/1qc8edat1gahfgt/chess-tournaments-2024-05.pgn/file

This is combined from TWIC, Chesscom Events, and Chess Results:
https://theweekinchess.com/twic
https://www.chess.com/events/results
https://chess-results.com/partieSuche.aspx?lan=1

Doubles and non-standard games are removed. Everything here qualifies in ChessBase as a strong game, i.e. at least 10 moves, and with ratings 2300+. Openings, evaluations, beauty scores, and novelty annotations are added.