A few days back, I hacked up a little minor mode for Emacs to use an LLM for spell checking. (Before you get a heart attack — I did not say “editing”, “getting ideas” or “research”: Just spell checking. Using an LLM for actual writing isn’t just a bad idea — it leads to abhorrent results.)
I’ve been running a random selection of blog posts I wrote for the Comico read through blog to see whether the thing works well — and… yes! But also no.
OK, positives first:
1) For simple typos, it’s very good indeed — it finds them all.
c) For word substitution errors, like “to” instead of “too”, it’s also very good.
II) For grammar errors, it’s more mixed. It’s good at, for instance, basic noun/verb agreement, so if I’ve written “they writes”, it fixes that. But if the noun clause is long and complex, it doesn’t always react. I’ve experimented with adding/removing an “s” from the verb in sentences like that, and it sometimes lets both versions through without comment.
⁵) Since I’m usually giving the LLM the complete blog post to proof-read, it knows what area I’m writing about, and so it’s able to correct me when I misspell names, too. I hadn’t even considered that, but that was a very nice surprise. So when I’m writing about comics and casually mention “the art looks a bit like Kriegstein”, it knows to correct that to “Krigstein”. This is something that traditional spell check can’t do at all. (And amusingly enough, running this article through the LLM doesn’t tag that “they writes” up there as an error, because it gets that that was an example of an error.)
So that’s all very good. So what’s the bad bits? Well, it’s an LLM, so it’s slow. That’s not a major problem, though, since I’m just using it to give a blog post a look-over to weed out the worst spelling mistakes. But I’ve added some code to strip out (longer) HTML fragments before sending the text over to the LLM, and this helps a bit.
The biggest problem is that an LLM is an LLM and will randomly decide to not follow the instructions in the prompt. Sometimes it adds random elements to the text here and there. Sometimes it says that I should fix a word from “identical” to “identical”. Sometimes it removes a paragraph. This doesn’t happen often — perhaps in 5% of the cases? But…
LLMs are unreliable! It cannot be! *gasp*
(Somebody speculated that this unreliability is a feature for the LLM companies — because you know that you may get bad results, you get extra happy the times you get a good result. The slot machine thing? I think that’s probably the case — some people get really addicted to their LLMs…)
Fortunately, these things can be programmed around — I’ve now added code to do sanity checks to the output from the LLM before using it. So if the LLM decides to go off piste, the package detects that, and then you can just do the query again.
It used to be that doing the same thing and expecting a different result was a sign of insanity, but with an LLM, that’s fine.
It’s a brave new world out there.
It’s also … odd … to type commands in Emacs and then they cost money.
Anyway, I’ve pushed these fixes to Microsoft Github, so there you go.