I’ve spent a day test-driving that Emacs minor mode for spellchecking using an LLM that I wrote the other day. I went through all the posts on the Total Eclipse blog (all 260 of them) to see whether the thing works or not.
And, yes, it works now, but it took some tweaking. Well, I rewrote the mode.
Twice.
My original approach was to give the LLM the entire text, and then ask it to return it to me verbatim, but with changed words tagged up specially. This worked 90% of the time, but it would randomly decide to do something else… and I was never totally sure that it actually returned the entire text. That approach was a bit too nerve-racking.
But Sacha Chua has done something along similar lines, and her approach is to ask the LLM to return just the changed parts:
That sounds like a much more responsible approach, so I redid my package to do pretty much the same… but no matter how I phrased things in the prompt, I couldn’t get the LLM to follow my instructions more than 50% of the time, really.
The hard thing to make it understand is that it should return a regexp that identifies the fragment uniquely, and that the word to be replaced should be inside a \(…\) group. But:
Some regexps would have zero grouping operators, and some would have several. And sometimes it would wrap the JSON in Markdown:
And worst of all, I could not get it to understand even once that it shouldn’t do this:
I.e., it seems like the LLM did the “regexps” by slapping a (…) around the term to be replaced, and then it put two backslashes before every parenthesis in the string. This is wrong.
So I redid the thing once again, but without regexps. Instead I asked it to return some JSON of the offending phrase (with enough context to uniquely identify it), and what it wants to replace it with:
And then I compute myself the actual difference between the phrases for markup purposes.
After this change, the LLM hasn’t screwed up the format once.
I guess what this really drives home for me is that the apparent enormous strides LLMs have taken the last six months for programming are mostly an illusion: You can now ask an LLM to whip you up, say, a Python script, and it chugs away at it, and then gives you a Python script that’s syntactically correct and mostly does what you asked.
But this isn’t because the LLMs now can generate correct syntax flawlessly. No, this is because the LLMs now have a sandbox they test-run all scripts before handing them over. So they discover that what they’ve generated is junk, and then iterate rapidly over it until it works.
But when I’m using the API here, it can’t check whether the regexps I asked it to generate are valid. It spits out something that’s close, but if it had its normal interactive harness, it would have tested the regexps and seen that they were invalid, and then fixed them.
So… it’s the LLM tooling that tricks us into believing that LLMs are pretty smart at computer stuff now. They aren’t, really.
Anyway, the Total Eclipse blog should now be 100% without typos! Woohoo! And since I’ve eyeballed every change, hopefully the LLM hasn’t changed too many instances of “important” to “load-bearing” without me noticing.
(One amusing thing I’ve noticed is that you can run the same article through the LLM several times, accepting the edits each time. As it finds fewer and fewer real things to complain about, it gets more and more nitpicky. But eventually it gives up and says “nothing to be changed”.)