NeuralStackSakana AI and researchers from the University of British Columbia, the Vector Institute, and the University of Oxford have published a paper in Nature describing The AI Scientist, a system built to automate much of the machine-learning research process. The open-access paper explains the system’s architecture, presents new scaling results, and examines both the potential and risks of AI-generated science.
The project began with a narrower prototype that received a code template, generated research ideas, ran experiments, and wrote a complete paper. Sakana AI also developed an Automated Reviewer to assess the resulting work. That first release was intended to show that end-to-end automation of machine-learning research was possible.
The later AI Scientist-v2 was given greater freedom to investigate broadly defined topics. Its developers submitted unedited, fully AI-generated papers to the blind human review process for the ICLR 2025 I Can’t Believe It’s Not Better workshop, with permission from the organizers. One manuscript exceeded the workshop’s average human acceptance threshold, although the team had decided in advance to withdraw it if accepted.
Under the hood, The AI Scientist generates research ideas, searches for and reads relevant literature, designs experiments, writes code, runs those experiments through parallelized agentic tree search, and drafts a paper in LaTeX. A foundation model with vision capabilities also provides feedback on figures, extending the system beyond text generation into parts of the experimental and editorial workflow.
To evaluate generated research at scale, the team created an Automated Reviewer that acts like an area chair by combining five independent reviews into a final decision using official NeurIPS guidelines. The researchers compared its judgments with thousands of human decisions from the OpenReview dataset and reported performance comparable to human reviewers. They then used the system to compare papers produced by different foundation models.
Why this matters
The results point to a potentially important shift in how AI research could be conducted. The quality of papers generated by The AI Scientist increased as the underlying foundation models improved, suggesting that stronger models and lower computing costs could make automated research systems more capable and more widely usable. If that trend continues, AI systems could increasingly help researchers explore large numbers of ideas and experiments rather than serving only as assistants for isolated tasks.
The publication also makes clear that passing peer review is not the same as producing consistently reliable science. The system can generate naive or underdeveloped ideas, struggle with methodological depth and complex code, and make mistakes such as inaccurate citations or duplicated figures. At present, it is limited to computational experiments, leaving its ability to contribute to other scientific fields unproven.
Sakana AI expects the playbook behind The AI Scientist to be adapted beyond machine learning and computational research. The Nature publication therefore marks both a milestone and a starting point: it demonstrates a working path toward more autonomous scientific discovery while leaving human oversight essential as the systems improve.