<p>Even if you had an LLM-based implementation of programmability, will it always work? Will it work for programs up to a certain size? How will you know when it stops working? Will it work for a different seed or at a different temperature?</p><p>The programmability fallacy, as I'll coin it, lies to us and tells us not to worry about those problems because we can tell LLMs to multiply small numbers by each other and sometimes get the right answer.</p>
