To understand the capabilities of AI in all these cognitive abilities, we propose a three-stage test protocol that shows the performance of the system in relation to human capabilities:
- Evaluate AI systems across a wide range of cognitive tasks covering each skill, using test sets designed to prevent data contamination.
- Collect demographic data for similar occupations from a representative sample of the adult population
- It maps the performance of each AI system relative to the distribution of human performance in each skill
From theory to practice
Defining these cognitive skills is an important first step, but we need more than a framework to measure progress. To make this idea work, we are launching a new Kaggle hackathon – “Measuring progress towards AGI: Cognitive skills”. The hackathon encourages the community to design tests for five cognitive skills where the test gap is greatest: reading, comprehension, attention, executive functions and social cognition.
Participants can use Kaggle's newly launched Community Benchmarks platform to build and test their analysis against a range of benchmark models.
We're offering a total prize pool of $200,000: $10,000 prizes for the top two submissions for each of the five songs, and $25,000 grand prizes for the top four submissions overall. Submissions are open from March 17 to April 16, and we'll announce the results on June 1. Go to the Kaggle website to start building.
