Anthropic has initiated a program to fund the development of innovative AI benchmarks capable of assessing the performance and societal impact of AI models, including its own generative model, Claude.
Announced recently, Anthropic’s initiative will provide grants to external organizations capable of creating benchmarks that can effectively measure advanced AI capabilities. Interested parties can submit applications on an ongoing basis for evaluation.
“Our investment in these evaluations aims to advance AI safety by providing essential tools that benefit the entire ecosystem,” Anthropic stated on its official blog. “Developing high-quality evaluations that address safety concerns remains a significant challenge, with demand surpassing current supply.”
AI benchmarking faces notable challenges today, particularly in accurately reflecting real-world user interactions and measuring modern generative AI capabilities against outdated benchmarks. Anthropic proposes creating rigorous benchmarks focusing on AI security and societal implications, including tools for tasks such as cyberattacks, potentially enhancing weapons, and manipulating people through techniques like deepfakes or misinformation.
The company also plans to develop an early warning system to identify and evaluate AI risks related to national security and defense, although specifics were not detailed in the blog post.
Anthropic’s program also aims to support research into benchmarks for tasks spanning scientific research assistance, multilingual conversation, bias mitigation, and toxicity self-regulation in AI.
To facilitate these goals, Anthropic envisions new platforms for subject-matter experts to develop evaluations and large-scale model trials involving thousands of users. The company has appointed a full-time coordinator for the program and is open to acquiring or expanding projects with scalability potential.
“We offer various funding options tailored to project needs and stages,” Anthropic mentioned, though specific details were not disclosed initially. “Teams will collaborate directly with Anthropic’s experts in red teaming, fine-tuning, trust and safety, and other relevant domains.”
While Anthropic’s effort to advance AI benchmarks is commendable, questions may arise about aligning funded evaluations with the company’s own AI safety classifications. Critics within the AI community might challenge assertions regarding catastrophic AI risks such as those related to nuclear weapons.
Anthropic hopes its initiative will set a new standard for comprehensive AI evaluation across the industry, fostering collaboration among various efforts striving for improved benchmarks. However, the extent to which independent initiatives will align with a commercial AI vendor’s objectives remains to be seen.
