“The planning thing in poems blew me away,” says Batson. “Instead of at the very last minute trying to make the rhyme make sense, it knows where it’s going.”

How is this surprising, like, at all? LLMs predict only a single token at a time for their output, but to get the best results, of course it makes absolute sense to internally think ahead, come up with the full sentence you’re gonna say, and then just output the next token necessary to continue that sentence. It’s going to re-do that process for every single token which wastes a lot of energy, but for the quality of the results this is the best approach you can take, and that’s something I felt was kinda obvious these models must be doing on one level or another.

I’d be interested to see if there are massive potentials for efficiency improvements by making the model able to access and reuse the “thinking” they have already done for previous tokens

Admittedly, that is a pretty big “if”. But yeah, if I manage to do it I certainly will!

Same boat here, recently discovered tana and its whole model is amazing. It’s fixing most of the things that bothered me a lot in Obsidian and Notion, respectively. I don’t want to go back to a service where I don’t have file-based control over my own data though, so now I’m seriously considering building something on my own that takes the mental model of tana, but implements it local-first based on regular files like Obsidian