François Chollet Co-Founds Nonprofit to Develop AGI Benchmarks
-
Amelia Cross
- January 8, 2025
- No Comments
François Chollet Co-Founds ARC Prize Foundation to Develop AGI Benchmarks
François Chollet, former Google engineer and influential AI researcher, is co-founding the ARC Prize Foundation, a nonprofit focused on creating benchmarks to evaluate artificial general intelligence (AGI). AGI refers to AI systems capable of performing most tasks humans can achieve.
The nonprofit will be led by Greg Kamradt, an ex-Salesforce engineering director and founder of AI product studio Leverage, who will serve as president and board member. Fundraising efforts for the foundation will begin later in January.
“[We] are growing … into a proper nonprofit foundation to act as a useful north star toward artificial general intelligence,” Chollet wrote on the foundation’s website. The ARC Prize Foundation will build on ARC-AGI, a benchmark developed by Chollet in 2019. ARC-AGI tests an AI’s ability to adapt to new tasks through puzzle-like problems requiring reasoning beyond its training data.
While ARC-AGI has revealed gaps in current AI systems’ reasoning abilities, future versions aim to narrow the gap between human and AI capabilities. Chollet noted that many models rely on brute force for high scores, but such strategies fall short of demonstrating true AGI.
The foundation plans to release a second-gen ARC-AGI benchmark in Q1 alongside a new competition, while also working on a third edition. Chollet and co-founder Mike Knoop aim to further AGI research by establishing academic networks and partnerships with frontier AI labs.
Notably, OpenAI CEO Sam Altman announced plans to collaborate with ARC-AGI on future benchmarks, but no updates on this partnership were included in the announcement. Despite debates over AGI’s definition and criticisms of ARC-AGI, Chollet remains committed to advancing tools for measuring progress toward human-level intelligence.
