You need to make sure you have the agents you want to compare selected in the Player Preferences before selecting this menu option.
Currently the app will let you run this comparison even if human players are selected, which will lead to an error. I'll fix this for the next release.
Ok. I thought it was going to compare all agents. I had MAST as both players when I first tried it. However, now that I have changed this, it still doesn't seem to do anything..
The Compare Agents menu option simply compares the currently selected agents on the loaded game.
The time take for this comparison to finish depends on how long the agents take. For some games, depending on the agents and thinking time selected, this may take a very long time....
If you only want to run a single playout, I would suggest just letting the two agents play against each other like normal. The only advantage of the compare agents option is that it allows many playouts to be run.
If you still want to use this menu option, can you please try running this comparisons with two MAST agents (1 second think time) on Tic-Tac-Toe and see if that completes for you?