Ran across this gem from HN about AI. I'll let it speak for itself: We've got a QA agent that needs to run through, say, 200 markdown files of requirements in a browser session... For the longest time we tried everything to get a prompt like the following working: "Look in this directory at the requirements files. For each requirement file, create a todo list item to determine if the application meets the requirements outlined in that file". In other words: Letting the model manage the high level control flow. This started breaking down after ~30 files. Sometimes it would miss a file. Sometimes it would triple-test a bundle of files and take 10 minutes instead of 3. An error in one file would convince it it needs to re-test four previous files, for no reason. It was very frustrating. There are people attempting to replace test suites with a prompt. An LLM... a probabilistic machine... to replace a test, a logic-based assertion. I don't even know where to begin.