how to spot a potential cheat

A few years ago, the chess website Chess.com temporarily banned US grandmaster Hans Niemann for playing chess moves online because the suspect side was suggested to him by a computer program. It had reportedly previously suspended his mentor Maxim Dlugy.

And at the Sinquefield Cup earlier this month, world champion Magnus Carlsen resigned without comment after playing a poor game against 19-year-old Niemann. He has since said it was because he believes Niemann has continued to cheat lately.

Another participant, Russian grandmaster Ian Nepomniachtchi, called Niemann’s performance “more than impressive”. While Nieman has admitted to sometimes cheating in previous online games, he has staunchly denied ever cheating in a live chess tournament.

But how does Chess.com, the world’s largest chess website, decide that a player has likely cheated? It cannot show the world the code used, otherwise would-be scammers would know exactly how to avoid detection. On the website it says:

Although Chess.com is prevented for legal and practical reasons from disclosing the full set of data, metrics and traces used to rank games in our fair play tool, we can say that the core of the Chess.com is a statistical model that outperforms the likelihood of a human player making an engine’s best decisions and the confirmed clean play of some of the greatest chess players in history.

Fortunately, research can shed some light on what approach the site might be using.

Humans vs AI

When the AI ​​company DeepMind developed the program AlphaGo, which could play the strategy game Go, it was taught to predict what movements a human would make from a certain position.

Predicting human motion is a supervised learning problem, the be-all and end-all of machine learning. With many examples of positions from human games (the data set) and an example of a human movement from each of those positions (the label), machine learning algorithms can be trained to predict labels at new data points. So DeepMind taught its AI to estimate the probability that a human would make a certain movement from a certain position.

AlphaGo defeated human rival Lee Sedol in 2017. One of the famous AI moves in the game was “Move 37”. As lead researcher David Silver noted in the AlphaGo documentary, “AlphaGo said there was a 1/10,000 chance that Move 37 would have been played by a human player.”

So, according to this machine learning model of human Go players, if you see a person playing Move 37, that would be proof that they didn’t come up with the idea themselves. But of course that wouldn’t be proof. Every person could take this step.

To be very sure that someone is cheating at a game, you have to look at a lot of moves. For example, researchers have looked at how many of a player’s moves can be analyzed together to detect anomalies.

Chess.com openly uses machine learning to predict what moves a human might make in a given position. In fact, it has different models of individual famous chess players and you can actually play against them. Presumably, similar models are used to detect fraud.

A recent study suggests that in addition to predicting how likely a person is to make a particular movement, it’s also important to consider how good that movement is. This is consistent with Chess.com’s statement that it evaluates whether moves beat the “confirmed clean play” of the greats.

But how do you measure which moves are better than others? In theory, a chess position is either “win” (you can guarantee a win), “lose” (the other player can), or “draw” (neither can), and a good move would be any move that doesn’t make your position worse. But realistically, while computers are much better at calculating and choosing future moves than humans, in many positions they can’t even say for sure if a position is winning, losing, or drawing. And they certainly could never prove it – a proof would generally require too many calculations and examining each leaf of an exponential game tree.

So what humans and computers do is use “heuristics” (gut guesses) to estimate the “value” of different positions – they estimate which player they think will win. This can also be viewed as a machine learning problem where the data set consists of many board positions and the designations of who won – which trains the algorithm to predict who will win from a given position.

Typically, the machine learning models used for this purpose think about the next few likely moves, consider which positions are accessible to both players, and then use “gut feelings” about those future positions to inform their assessment of the current position.

But who wins from a certain position depends on how good the players are. So, the model’s assessment of a particular game depends on who played the games that made it into the training dataset. When chess commentators talk about the “objective value” of different positions, they usually mean who is likely to win from a particular position when both sides are played by the very best chess AIs. But this measure of value isn’t always the most useful when considering a position that a human player must end up taking. So it’s not clear what Chess.com (or we) should consider a “good move”.

Cheating at chess and making a few moves suggested by a chess engine may not even help me win. These moves could set up a brilliant attack that I would never think of, so I would waste it unless I asked the chess engine to play the rest of the game for me. (Lichess.org tells me I’ve played 3,049 blitz games at the time of writing, and my not very good ELO rating of 1632 means you can expect me to be missing good tactics left and right.)

Detecting fraud is difficult. If you play online and wonder if your opponent is cheating, you can’t say for sure – because you haven’t seen millions of games with people played with radically different styles. It’s a problem where machine learning models trained on massive amounts of data have a major advantage. Ultimately, they can be vital to the ongoing integrity of chess.