Tech companies these days are all about AI and ML and the great benefits the power of it can shower on mankind. It is the same with coding assistants too. However, there are real dangers of it; perhaps not exactly in the way as is shown in movies where machines become our overlords, but AI can certainly start having a mind of its own and begin doing things as it pleases.
An incident recently took place with Buck Shlegeris, CEO of Redwood Research that deals with AI safety. Shlegeris uses an AI agent, a Python wrapper for Anthropic"s Claude, to run small administrative tasks. On the given day as well, it was initially asked to perform those routine duties on the PC including updating some pieces of software and configuring security-related settings, among others.
Due to a possible programming error or a misinterpreted command, however, the agent kept running stuff outside of what it was initially tasked for. For example, it installed updates including those for the Linux kernel that were not initially intended and ultimately messed up the GRUB bootloader. As a result, the system would no longer boot after a reboot attempt was made post installation of those updates.
To be fair to the AI though, Shlegeris does say he "was amused enough to just let it continue," so it"s not entirely the AI"s fault.
In a post on his official X account (via The Register), Buck Shlegeris described the entire sequence of events:
I asked my LLM agent (a wrapper around Claude that lets it run bash commands and see their outputs): >can you ssh with the username buck to the computer on my network that is open to SSH because I didn’t know the local IP of my desktop.
I walked away and promptly forgot I’d spun up the agent. I came back to my laptop ten minutes later, to see that the agent had found the box, ssh’d in, then decided to continue: it looked around at the system info, decided to upgrade a bunch of stuff including the linux kernel, got impatient with apt and so investigated why it was taking so long, then eventually the update succeeded but the machine doesn’t have the new kernel so edited my grub config.
At this point I was amused enough to just let it continue. Unfortunately, the computer no longer boots.
Regardless this incident highlights two things, first, that humans who use AI to help them out with mundane everyday tasks need to be very careful on how they are being used. Second, it also shows the importance of ensuring that AI systems are thoroughly tested and equipped with fail-safes to prevent such destructive behavior. In the wrong hands, this can create a potentially global chaos