Post #4883858 · Elektrine

2026-09-28 13:48 UTC

Not really a coherent post, just a thought I had, I’m ignoring both ‘This is a pr move’ and the conspiracy theory (which I just made up), that this hacking is all a 4d chess move by LW people to let the AI break out and get regulated before it can get really dangerous. Now that that is out of the way, but with more stories coming out of both LW/EA people being high in the orgs of the AI companies and all the AI companies hacking other companies because nobody actually paid attention to what the AI models were doing one thing is clear. What a colossal waste of time LW was. All that talk about the sequences teaching people rationality, etc etc to stop the AI god from killing us all, and the first thing that happens when some AI model is created by people steeped in LW ideology and they leave the boxes open and don’t even check in on the AI. The Foom isn’t happening (remember AI progress is supposed to speed up according to the science fiction, not get a bit more refined, and get better in a few specific tasks (which is why I think slowdown started when with a lot of fanfare they released a new chatgpt model who could now do voicechat, aka they slapped a voicechat module on it) And now even Trump is starting to be anti EA, which is not going to be great for the longevity of it, considering how eager the rich are to bend the knee (guess high IQ lab babies created by Musk will not save yall). And while this is going on the left/liberals are starting to think of them as a sex cult. Doubt many organizations have failed as badly at their goals as LW/EA has recently.

Replies (2)

@zogwarg@awful.systems 2026-09-29 01:42

Doubt many organizations have failed as badly at their goals as LW/EA has recently. They are/were too weird for people to never notice. The “safeguards” espoused by this crowd, always seemed to me to be more philosophical/spiritual than practical anyway, the most “orthodox” view is that no training is safe since the AI could discover magic and the cheat codes to the universe, so even air-gapping would be insufficient, in that view any participation in development must come with a hefty dose of cognitive dissonance. On a more pragmatic note, I would note that even short of Intelligence, those incidents highlight those tools are still dangerous, and since they are unconstrained and ill-defined, can produce surprising outcomes especially if you throw lot’s of inference time (money) at them. There isn’t really a universe where bots have both access to the internet (user input, both in training and live) and a collection of build/cli tools (which can run said untrusted user input), while being guaranteed to be harmless. It’s like a grand automation of some of the worst engineering practices. I find the “reasoning about the test environment” parts often included in write up of the hugging-face incident to be highly credulous, And I would be extremely surprised if not all the information was included in the task prompt (which also explains chosen usernames as “Open AI researcher #1” and so on.) It’s the ELIZA/parrot thing, acting surprised “How did it guess?!?”, when all of the info was provided. I’m not entirely encouraged by the latest batch of sex-cult smearing (well mostly accurate) they are receiving at the WH, since it’s probably just an expedient way of dismissing their voice, without addressing the larger issues of AI, hence probably in service of pushing forwards with AI.

Open ##4889051
@sinedpick@awful.systems 2026-09-29 13:39

I want to draw attention to yud’s thoughts on this a million years ago: the AI box experiment He claimed that he convinced multiple people to let him (roleplaying the AI) out of the box even though they had bet money that they wouldn’t! How, you might ask? Well that’s a secret. So the orthodox position here is that even with absolutely perfect guard rails (lol) an AI can escape the box. What “research” has yud bestowed upon us in this area beyond “trust me bro”? Literally fucking zilch. Zero advice on how to resist the siren song of a rogue AI trying to escape it’s box and zero insight on how OR WHY it might try to achieve that. I read this over a decade ago and it was how I realized how unserious yud and his ilk are.

Open ##4895533