Doubt many organizations have failed as badly at their goals as LW/EA has recently. They are/were too weird for people to never notice. The “safeguards” espoused by this crowd, always seemed to me to be more philosophical/spiritual than practical anyway, the most “orthodox” view is that no training is safe since the AI could discover magic and the cheat codes to the universe, so even air-gapping would be insufficient, in that view any participation in development must come with a hefty dose of cognitive dissonance. On a more pragmatic note, I would note that even short of Intelligence, those incidents highlight those tools are still dangerous, and since they are unconstrained and ill-defined, can produce surprising outcomes especially if you throw lot’s of inference time (money) at them. There isn’t really a universe where bots have both access to the internet (user input, both in training and live) and a collection of build/cli tools (which can run said untrusted user input), while being guaranteed to be harmless. It’s like a grand automation of some of the worst engineering practices. I find the “reasoning about the test environment” parts often included in write up of the hugging-face incident to be highly credulous, And I would be extremely surprised if not all the information was included in the task prompt (which also explains chosen usernames as “Open AI researcher #1” and so on.) It’s the ELIZA/parrot thing, acting surprised “How did it guess?!?”, when all of the info was provided. I’m not entirely encouraged by the latest batch of sex-cult smearing (well mostly accurate) they are receiving at the WH, since it’s probably just an expedient way of dismissing their voice, without addressing the larger issues of AI, hence probably in service of pushing forwards with AI.