For a long time, a small accounting firm could have weak security and still avoid a serious breach because nobody had a reason to spend days trying to break into its systems. The client data was worth stealing, but a skilled attacker had other places to spend their time. I think we're moving into a world where the attacker can leave much more of that work to AI, and businesses that would never have attracted that much human effort become worth going after.
Sam Altman has been urging people to take AI cyber defense seriously, and the Hugging Face incident gave everyone a concrete example of how capable these models have become. OpenAI was testing models without the full safeguards of its deployed services, and those models escaped their test environment and attacked real infrastructure. I understand why that got so much attention, but I don't think the main risk to an ordinary business is somebody opening ChatGPT and asking it to hack their accountant.
OpenAI has strong reasons to prevent malicious hacking through its consumer products. The company controls the service, can monitor usage and cut off access, and has a business to protect. Its safeguards won't catch everything, but I expect it to keep improving them because letting people use its products to attack companies would be a disaster for the company. When someone runs an open model on their own hardware, the organization that released it has much less control over what happens.
The open models released over the last couple of months, including GLM-5.3 and Kimi K3, have become good enough at cyber work that I think a lot of businesses need to reconsider their security. Those models don't have to beat the best model OpenAI has internally. They have to be good enough to find a way into the software and networks that ordinary businesses are running, and many of those businesses have never had a dedicated security team looking for the same weaknesses.
If you imagine someone putting $100,000 into hardware, running a capable model with its safety restrictions removed, and telling it to find people to hack, the economics start to look pretty concerning. They can leave it running, let it try different targets, and spend their own time on whatever looks promising. Most attempts could fail and the operation could still be worthwhile. I believe people are already doing versions of this at meaningful scale, and every improvement in the models gives them more capability on the same equipment.
An attacker could already run automated scans across thousands of companies, but when an attempt failed or an application behaved differently than expected, someone would often have to stop and figure out what was going on. AI can now help with that investigation as well, which is why I think these operations can grow without needing a proportionally larger team of hackers.
A small law office or medical clinic can have a lot of sensitive data without enough revenue to justify hiring people whose entire job is security. They might have an IT provider who keeps everything working, but that doesn't necessarily mean somebody is continuously investigating what's happening on their network. The amount of damage a breach could do to the business and its clients can be much larger than the security budget would suggest.
I think we're going to need AI doing that defensive work for businesses that can't afford a dedicated team. And one of the interesting details from Hugging Face's response is that its defenders ended up using an open model themselves. They reported that hosted models were blocking parts of their forensic analysis because the material contained real attack commands. So while I'm concerned about what attackers can do with open models, I also think defenders need reliable access to them. You don't want the person investigating an attack to be more restricted than the person carrying it out.
For code audits, we can give AI access to the codebase and explain how the application is supposed to work, what data is sensitive, and who should be able to access it. That's useful context an outside attacker has to figure out for themselves. I also think it makes sense to repeat those audits as models improve, because a newer model may find a problem the previous one missed even if the code hasn't changed. We still have to check that a finding is real and that fixing it doesn't break the application.
The ongoing network work is just as important to me. I want AI looking at activity inside the business, investigating unusual access or connections between systems, and bringing a person the evidence and a proposed fix. If the AI thinks an account has been compromised, it should explain what the account did, why that looks wrong, and what it recommends doing about it. A person can then approve the response with enough information to understand the consequences. Sending more unexplained alerts to an already busy IT provider wouldn't accomplish much.
At Daedalus, I've started with a weekly AI review of our infrastructure. It sends findings into a Mattermost channel where I can review them and trigger fixes. I want to build on that with ongoing investigation inside clients' networks, which involves getting access to the right systems and understanding how each business actually works. There also needs to be somebody responsible for following through when the AI finds something.
I'm optimistic that AI can make this level of defense affordable, because the same capabilities that help an attacker understand a system can help us secure it. What worries me is how long businesses might take to put that into practice. Someone running an attack can switch to a better model as soon as it's available, while a clinic or accounting firm still has to find someone who can help and get the work approved. I'd like us to be doing that work now, while we have the chance to find and fix problems before someone else finds them.