If you truly want to understand the OpenAI o3 and ARC-AGI debacle this is the post I recommend.
Professor Melanie Mitchell provides history, context and theory, backed with extensive experience. Mitchell is the author or editor of six books and numerous scholarly papers in the fields of artificial intelligence, cognitive science, and complex systems.
In her post you get to understand…
- the purpose of the ARC-AGI competition, why she likes the ARC domain as a challenge for AI systems, and how the use of ”AGI” muddies the waters
- the importance of inference-time compute
- the differences between Deep Blue (the chess-playing computer) and AlphaGo
- what OpenAI never tells us and experts can only speculate around
- how humans are still demonstrably better at abstraction and reasoning
- and why we need to avoid Goodhardt’s law when developing a measure of intelligence
Also note that o3 is not eligible for testing on ARC’s fully private set, because o3 requires access to the internet. To be eligible for the Arc Prize, the winner’s program must run on the test set in at most 12 hours on the competition’s servers, with no internet access, and the code has to be open-sourced at the end of the competition.
As Mitchell describes, all of the ways the ARC-AGI competition has been approached negate the core assumptions of ARC with regards to amount of training, the required number of examples of grid transformations, and the computational resources used.
”Indeed, o3’s performance on ARC is quite amazing, even though it violates the assumption that solving these tasks shouldn’t require huge amounts of computation at inference time. But I’m most curious about an even more pressing question: Is o3 (or any of the other current winning methods) actually solving these tasks using the kind of abstraction and reasoning the ARC benchmark was created to measure?”