2026-05-25 14:23 UTC
@zwarich@hachyderm.io I would expect there to be an inverse correlation between the amount of implicit structure and the ability of a model to correctly generate the structure, without a lot of specific training. Seems like at some point you’d start spending more tokens fixing syntax errors from the first attempt
Replies (2)
-
@joe@f.duriansoftware.com 2026-05-25 16:38
@zwarich@hachyderm.io seems like only a matter of time before they rediscover factor here
-
@zwarich@hachyderm.io 2026-05-25 16:51
@joe@f.duriansoftware.com It’s funny how many of the tricks used to make language models work well on natural language reintroduce some of the limitations of humans, e.g. with subject-verb dependency distance, even though an attention mechanism with a gigantic context doesn’t have the same working memory limitations as humans.