18

It is well known that both hardware and software contributed to the total dominance of computers. Has there been an experiment to run a very modern program (say Alpha Zero) on a museum computer (of course, a memory issue can happen, so don't abuse the Zuse for your amuse) or vice versa, a museum program on a very fast computer? If yes, with what results (hard- or software more important)?

Improve this question
9
  • 2
    I see compatibility problems in running such experiments Commented Jan 3, 2023 at 19:44
  • 3
    It's the software. In computer science we generally say that we would rather run today's algorithms on yesterday's computers than vice versa. Chess shouldn't be an exception. Commented Jan 4, 2023 at 9:00
  • 1
    @Hauptideal Hmm, hardware acceleration is easily an order of magnitude (in terms of execution time) for modern DL models at least. I'd say software is always on the forefront of computing progress, and you can poorly run something advanced today knowing that tomorrow it will run well. But hardware still imposes limits on this progress, you can feasibly be two steps ahead, but not two thousand. We would not have Deep Blue if all we had was PDP-11. Commented Jan 4, 2023 at 16:51
  • 1
    @Lodinn that's all just constant factors. I'm taking about real Big-O improvements! And yet while something like this didn't happen in chess, we have much much better evaluations today than we had before. Commented Jan 4, 2023 at 19:13
  • 2
    I’m voting to close this question because this is not about chess, but about software vs hardware performance improvements. Software Engineering or the main site is the place to go. Further, the question is quite opinion based. Commented Sep 18, 2024 at 7:57

7 Answers 7

Reset to default
25

The answer is software is more important. You can test this yourself actually. Get your favorite top engine today and your favorite top engine from 10 years ago, and run both of them against each other at time odds. This simulates superior hardware (since it yields a higher number of positions searched).

This was done at 50-1 NPS (nodes per second, a measure of how fast the hardware is) odds here. Komodo 8 (a top 2014 engine), played on a smartphone against Shredder 10 (top 2006 engine) on an i7 desktop. Komodo won 5-1. Not a large sample size by any means, but clear enough given auxiliary data.

See also How does engine strength scale with hardware?, which in conjunction with rating lists played on equal hardware gives an idea of where hardware effects might dominate over software.

Improve this answer
7
  • 3
    Interesting. You'd run into problems running modern engines on true museum-piece hardware, though, like when memory limitations prevent modern engines from keeping their working set in RAM (especially deep learning models?). Or on truly ancient machines, when the depth they can see (in the time limit you give them) is so much lower than their algorithms / heuristics were designed and tuned for, they might make huge blunders. Maybe even worse than simple chess-playing programs written for such machines that use simpler heuristics. Commented Jan 4, 2023 at 8:11
  • Yes software by far. More so once realizing said software isn't, but could be, (re)written to run on toasters. Commented Jan 4, 2023 at 12:46
  • 1
    I wonder how well a chess engine could be programmed to perform if constrained to various historical classes of machinery, including e.g. the Fidelity Chess Challenger (a Fairchild F8 I think), a Atari 2600 with first-generation cart (6502 with 4K of code and 128 bytes of RAM) or later-generation cart (32K of code and 256 bytes of RAM), an Apple II+ (6502 with 48K RAM and 140K floppy), a 640K IBM PC using RAM only, a 640K IBM PC with up to 5MB of hard drive space, etc. An Apple II wouldn't be competitive with Stockfish, but modern knowledge might allow vastly better play than any programs... Commented Jan 5, 2023 at 16:09
  • ...written for that platform in the 20th Century. Commented Jan 5, 2023 at 16:09
  • 1
    One exercise in historical chess comparisons has been done by hippke. See also: SAT solving, LP/MILP, CNNs, SAT/chess/LP/ML, SAT again, LP. Commented Jan 14, 2023 at 20:14
12

I develop mobile chess apps to the App Store. The very first version of the apps were built on the iPhone 3GS. That was many years ago, but even then the apps could play at the GM chess level.

There is a rating list for Android chess engines:

You can then compare the list with more powerful hardware configurations:

No rating list exists for iOS due to technical limitations.

According to the JCER rating list, there is a big gap. Stockfish on Android is 3155 while the PC version is 3734.

Improve this answer
4

Note that there is a massive jump for the software side with the introduction of neural networks like Alpha Zero. Software before that required a large amount of clever humans programming it but these humans could do that with low end hardware. Neural networks need in comparison much less human programming time but require lots of computation power once to train them. After the training is done they can be run on worse hardware than pre neural network software and still play much better.

This makes numerically comparing pre neural network to post neural network software difficult because it is not clear what exactly you want to compare. One could run a modern chess software on say year 2000 hardware and it would trash any software available back then but I don't know how feasible it would have been to train a modern chess software with year 2000 hardware. A chess software probably can't afford to rent a super computer for a month of time for training.

Improve this answer
4
  • The obvious way to do it to me would be to run the new software at the same nps. For example, if a top engine in 2000 evaluates 2 million positions per second on the hardware of the day, we could also run a top engine of today at 2 millions positions per second. If that's not feasible, we could run a top engine today at time odds so that it searches the same number of positions as the old engine. It's not ideal - the complexity of the evaluation functions affect nps - but it should be close enough. Commented Jan 5, 2023 at 13:12
  • 1
    What would be the minimum amount of memory and CPU speed to make a neutral-network-designed chess bot usable? Would a desktop computer from 1992 (e.g. 80486 with 4MB of RAM and 120MB of hard drive storage available) be able to benefit? What about one from 1980 (e.g. 6502 with 48KB of RAM and 140K of floppy-drive storage)? Commented Jan 5, 2023 at 20:57
  • @supercat This is a good question but I unfortunately don't know enough about hardware requirements of neural-network software to provide a meaningful answer. Commented Jan 6, 2023 at 7:20
  • @supercat: I already did a quick google to find the space specs of Alpha Zero, but with no avail. (They surely are in the original SCIENCE paper.) Some data for Leela: lczero.org/play/troubleshoot Commented Jan 6, 2023 at 9:14
3

I don't see how any experiment would decide the question of "importance".

If you tried to run modern software on ancient hardware, it would fail, because modern software is designed to take advantage of modern hardware.

If you ran ancient software on modern hardware, it would do whatever it did before, only faster. Would it beat modern software running on modern hardware? Almost certainly not, because it wasn't written to take advantage of the hardware power available.

But consider: a lot of the improvement in computer chess is due to the existence of databases of openings and endgames. There's no clever software behind that, just availability of cheap storage. So do you attribute that to software or hardware advances?

Improve this answer
1
  • Under testing conditions with the same openings for both engines and no endgame tablebases, Stockfish has still gotten much stronger, and testing done with Stockfish has shown that endgame tablebases have a minimal impact on playing strength (I don't think opening databases have been tested much, but I would expect their effect to mainly be saving time on the opening, unless the game is using an extremely short time control where precalculated moves could be significantly higher quality than live ones) Commented May 24 at 1:00
1

"Turochamp", the original chess-playing programme from 1948 by Alan Turing and David Champernowne, was executed manually for a game in 1952, since hardware at the time was deemed incapable of running the software proper. It took about 30 minutes per move, and lost to Alick Glennie after 29 moves.

In 2012, for the Turing anniversary celebrations, the programme was reimplemented on contemporary hardware, losing to Garry Kasparov after 16 moves. Kasparov explained that the basic problem was that Turochamp is "only two ply", that is, it cannot look ahead very far and in particular could be deceived by a sufficiently skilled player who was aware of this limitation, regardless of the speed at which it would produce its moves.

As a caveat, the 2012 programme was not the same as the original, as the authors didn't preserve all details of the algorithm, and may not have implemented it thoroughly for the 1952 game in any case. Nonetheless, this demonstrates a long time gap, and that even with much, much better hardware, the algorithm is still not that capable.

Improve this answer
1
  • But given modem hardware it was allready know in 1948 how a "deap" search system could be implemented. Commented Sep 18, 2024 at 12:11
0

It's mainly software, especially at long time controls. There is a 1000 Elo rating point gap between Chess Genius 3 (a top program of 1994) and the latest stockfish on the same android or iOS device.

According to Wikipedia: https://en.wikipedia.org/wiki/ChessGenius, Chess Genius achieved an Elo rating point of 2795, which is about 1000 short of the 3641 of Stockfish, according to Wikipedia: https://en.wikipedia.org/wiki/Stockfish_(chess)

Improve this answer
2
  • 1
    I found this answer interesting but am left wondering what the source is that supports the claim. Commented Sep 18, 2024 at 14:59
  • The 2795 is a performance rating in game in 25 minutes, not a rating in slow time controls .SSDF gives a rating of just under 2500 for chess genius on a recent Android device my original 1000 Elo gap stands Commented Sep 28, 2024 at 22:27
0

Chess engines search the chess game space.

The chess game space is immense. So immense in fact, that running the same engine on faster hardware will yield almost no advantage. You can't win against chess by searching faster. The search must get better instead.

All the engine progress is in narrowing the tree of games that have to be searched. Newer/better engines search smaller game spaces. The gains from making the game space smaller for the engine to consider beat any hardware speed-up gains.

In computer science, this is known as "linear solutions can't solve exponential problems". To get any credible gains against an exponential problem (like chess), the algorithm needs to get better. Hardware speed is the "linear" scaling factor. Algorithmic improvements "grow" in effect much faster than a straight line plot.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.