A good way of thinking about companies like Proton or Anthropic that proudly declare they're using LLMs to code is that they've declared "we're replacing all of our junior engineers with unpaid interns from a temp agency." The unpaid interns have photographic memories and have memorized all of the code on github, but they're still pretty clueless and clumsy, and they also have zero loyalty to the company, zero ability to learn from mistakes, and zero understanding of or interest in the actual purpose of what they're building (actually, this comparison is unfair to interns on average, think of the least motivated and curious intern you've ever worked with).
The remaining staff at these companies become managers, but in the most frustrating possible way, since the agents they're managing constantly mess stuff up and have been made incapable of learning from their mistakes. Their working memory is actually pretty limited too, so if your instructions get too complicated, they start forgetting the first part of what you said. You can try to compensate for their inability to learn by writing down more instructions in text files for them to read, but the issue with their working memory makes this a losing strategy in the long run. You can try to carefully control which reminders of what not to do they read based on what parts of the code they're working on, but you don't really have good tools to control that either (such tools could be built, and would have been already if those in charge weren't so incompetent and mesmerized by their own shitty mirages).
So when things like this Cursor attack happen (or whenever the next big Proton vuln hits; it's clearly coming soon based on their dev practices) companies will say "humans make the same kinds of mistakes" or "human programmers might have written that vulnerability too." They're not exactly wrong, but they're eliding the fact that teams of temp interns make mistakes at rates orders of magnitude higher than teams of experienced security experts, and having fired the later to hire the former is a choice that everyone buying your product should pay attention to, whether the former comes in the form of actual incompetent interns or LLM agents.
#AI #LLMs #Proton #Anthropic #Cursor
