28 Jun 2024 Search

Now that AI can beat a Turing test by bullshitting, what’s the next test? In Prediction Market Trading as an LLM Benchmark, Jesse Richardson suggests that setting up an LLM to trade on a prediction market (e.g. Polymarket, which is the platform I’ll talk about here) could be a particularly strong benchmark with a number of desirable properties. Scott Alexander also suggests prediction markets as a useful challenge for bots.

Seems like a good idea. The best part about prediction markets as a benchmark is scaling. One skilled prediction market trader can end up on the winning side of trades with a large number of low-skill traders, human or bot. LLM benchmarks that depend on evaluating generated text are much harder to scale, and it’s usually easier to bullshit than to detect bullshit. To make a market-based pass/fail test, give the bot a stake and charge it interest on its stake and rent for the computing resources it uses. A bot passes the test if it can stay solvent for the agreed-upon time in a market with human traders. Even if bots can’t pass the test, they might still have their uses, since they can help add liquidity to corporate prediction markets and incentivization markets. (More: boring bots ftw)

In practice this market test would not be a pure prediction benchmark, but would have some text generation aspects, too. A bot would get a substantial advantage by explaining its trades afterwards—talking up its positions, and telling other traders why they should pay more than the bot did. But the bot’s ability to explain its actions in a persuasive way does not have to be tested just by looking. It can be evaluated indirectly by looking at how well the bot can persuade other traders.

The big problem with building an open ecosystem for AI is the cost of benchmarking. Even when it is possible to train a model at low cost, evaluating that model depends on either hiring large numbers of human reviewers, or paying for access to a larger model. In order to get to the own your own AI stage, the benchmarking budget problem needs to be addressed early on. Integrating an incentivization market with existing open-source collaboration infrastructure (Pinfactory connects to a GitHub project pretty easily, just saying) could be a good start. More: boring bots ftw

Update 26 Apr 2026: Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets by Jaden Zhang, Gardenia Liu, Oliver Johansson, Hileamlak Yitayew, Kamryn Ohly, Grace Li. Prediction Arena demonstrates that real-world evaluation with real consequences provides insights that cannot be obtained from synthetic benchmarks. Over the full 57-day evaluation, Cohort 1 models exhibited final returns ranging from −16.0% to −30.8% — a spread driven not by market luck but by systematic differences in initial prediction accuracy, risk discipline, and (in Phase 2) the decision of whether to continue trading at all. These differences are not visible on standard NLP benchmarks, illustrating the measurement value of high-stakes, real-money evaluation….Prediction Arena operationalizes the distinction between information gathering and information synthesis in a way that static bench- marks cannot.