According to Sakana AI, in one run, it edited the code to perform a system call to run itself.

“This led to the script endlessly calling itself. In another case, its experiments took too long to complete, hitting our timeout limit. Instead of making its code run faster, it simply tried to modify its own code to extend the timeout period.”

Sakana provided two screenshots of example code generated by the AI model, and the 185-page AI Scientist research paper delves deeper into what they call “the issue of safe code execution.”

While the AI Scientist’s behaviour did not pose immediate risks in the controlled research environment, these instances highlight the importance of not allowing an AI system to run autonomously in a non-isolated environment.

The researchers emphasise that AI models do not need to be “AGI” or “self-aware” (both hypothetical concepts at present) to be dangerous if allowed to write and execute code unsupervised.

Such systems could inadvertently break existing critical infrastructure or potentially create malware.

These findings underscore the need for stringent safety measures and supervision when deploying AI systems capable of autonomous code execution.