The hallucinations come from the llm losing context. The window is only so large.