Spellchecking Total Eclipse with an AI

I’ve spent a day test-driving that Emacs minor mode for spellchecking using an LLM that I wrote the other day. I went through all the posts on the Total Eclipse blog (all 260 of them) to see whether the thing works or not.

And, yes, it works now, but it took some tweaking. Well, I rewrote the mode.

Twice.

My original approach was to give the LLM the entire text, and then ask it to return it to me verbatim, but with changed words tagged up specially. This worked 90% of the time, but it would randomly decide to do something else… and I was never totally sure that it actually returned the entire text. That approach was a bit too nerve-racking.

But Sacha Chua has done something along similar lines, and her approach is to ask the LLM to return just the changed parts:

That sounds like a much more responsible approach, so I redid my package to do pretty much the same… but no matter how I phrased things in the prompt, I couldn’t get the LLM to follow my instructions more than 50% of the time, really.

The hard thing to make it (I’m using Sonnet 5) understand is that it should return a regexp that identifies the fragment uniquely, and that the word to be replaced should be inside a \(…\) group. But:

Some regexps would have zero grouping operators, and some would have several. And sometimes it would wrap the JSON in Markdown:

And worst of all, I could not get it to understand even once that it shouldn’t do this:

I.e., it seems like the LLM did the “regexps” by slapping a (…) around the term to be replaced, and then it put two backslashes before every parenthesis in the string. This is wrong.

So I redid the thing once again, but without regexps. Instead I asked it to return some JSON of the offending phrase (with enough context to uniquely identify it), and what it wants to replace it with:

And then I compute myself the actual difference between the phrases for markup purposes.

After this change, the LLM hasn’t screwed up the format once.

I guess what this really drives home for me is that the apparent enormous strides LLMs have taken the last six months for programming are mostly an illusion: You can now ask an LLM to whip you up, say, a Python script, and it chugs away at it, and then gives you a Python script that’s syntactically correct and mostly does what you asked.

But this isn’t because the LLMs now can generate correct syntax flawlessly. No, this is because the LLMs now have a sandbox they test-run all scripts before handing them over. So they discover that what they’ve generated is junk, and then iterate rapidly over it until it works.

But when I’m using the API here, it can’t check whether the regexps I asked it to generate are valid. It spits out something that’s close, but if it had its normal interactive harness, it would have tested the regexps and seen that they were invalid, and then fixed them.

So… it’s the LLM tooling that tricks us into believing that LLMs are pretty smart at computer stuff now. They aren’t, really.

Anyway, the Total Eclipse blog should now be 100% without typos! Woohoo! And since I’ve eyeballed every change, hopefully the LLM hasn’t changed too many instances of “important” to “load-bearing” without me noticing.

(One amusing thing I’ve noticed is that you can run the same article through the LLM several times, accepting the edits each time. As it finds fewer and fewer real things to complain about, it gets more and more nitpicky. But eventually it gives up and says “nothing to be changed”.)

Some LLM spell check experiences

A few days back, I hacked up a little minor mode for Emacs to use an LLM for spell checking. (Before you get a heart attack — I did not say “editing”, “getting ideas” or “research”: Just spell checking. Using an LLM for actual writing isn’t just a bad idea — it leads to abhorrent results.)

I’ve been running a random selection of blog posts I wrote for the Comico read through blog to see whether the thing works well — and… yes! But also no.

OK, positives first:

1) For simple typos, it’s very good indeed — it finds them all.

c) For word substitution errors, like “to” instead of “too”, it’s also very good.

II) For grammar errors, it’s more mixed. It’s good at, for instance, basic noun/verb agreement, so if I’ve written “they writes”, it fixes that. But if the noun clause is long and complex, it doesn’t always react. I’ve experimented with adding/removing an “s” from the verb in sentences like that, and it sometimes lets both versions through without comment.

⁵) Since I’m usually giving the LLM the complete blog post to proof-read, it knows what area I’m writing about, and so it’s able to correct me when I misspell names, too. I hadn’t even considered that, but that was a very nice surprise. So when I’m writing about comics and casually mention “the art looks a bit like Kriegstein”, it knows to correct that to “Krigstein”. This is something that traditional spell check can’t do at all. (And amusingly enough, running this article through the LLM doesn’t tag that “they writes” up there as an error, because it gets that that was an example of an error.)

So that’s all very good. So what’s the bad bits? Well, it’s an LLM, so it’s slow. That’s not a major problem, though, since I’m just using it to give a blog post a look-over to weed out the worst spelling mistakes. But I’ve added some code to strip out (longer) HTML fragments before sending the text over to the LLM, and this helps a bit.

The biggest problem is that an LLM is an LLM and will randomly decide to not follow the instructions in the prompt. Sometimes it adds random elements to the text here and there. Sometimes it says that I should fix a word from “identical” to “identical”. Sometimes it removes a paragraph. This doesn’t happen often — perhaps in 5% of the cases? But…

LLMs are unreliable! It cannot be! *gasp*

(Somebody speculated that this unreliability is a feature for the LLM companies — because you know that you may get bad results, you get extra happy the times you get a good result. The slot machine thing? I think that’s probably the case — some people get really addicted to their LLMs…)

Fortunately, these things can be programmed around — I’ve now added code to do sanity checks to the output from the LLM before using it. So if the LLM decides to go off piste, the package detects that, and then you can just do the query again.

It used to be that doing the same thing and expecting a different result was a sign of insanity, but with an LLM, that’s fine.

It’s a brave new world out there.

It’s also … odd … to type commands in Emacs and then they cost money.

Anyway, I’ve pushed these fixes to Microsoft Github, so there you go.

English is such an impoverished language

That’s what I say, jokingly, every time I discover that English doesn’t have that particular word I’m looking for to distinguish between two natural concepts.

And it’s a joke, because of course English is a huge language, what with it swallowing up several other languages wholesale and then just misspelling all the words a lot. (It’s something Norwegians seriously say about Norwegian, though.)

But, like, for instance — many languages have words to distinguish between “a 24 hour period” and “a day”, and English doesn’t, so that’s awkward. And Norwegian has different words for “acting suspicious” and “being suspicious of others”.

English is such an impoverished language, man!

(Here’s where you pull out your OED and say “but see? here’s one!” and point to a word that has three hits on Google as evidence that English really does have that word.)

Joking aside, I there’s one area where English really does seem to have a lack of words/concepts that do make a difference — where you can see people are trying to say something about an issue, but the words to describe the issue just aren’t there. Not to go all Whorfian on you, but the other week I read an article that just made me go *GNGNGNG* because I was agreeing with everything it was trying to say, but it seemed to lack the concepts of what it was describing.

It was this article about Sally Rooney’s novel Normal People, and it describes the book as “romantasy”:

‘Right,’ you say. ‘So that’s surely not Normal People, which is set in 2010s Ireland. Not a dragon or a well-endowed faerie prince in sight.’

Which, I mean. I don’t know about that last part. Normal People certainly gives us to suspect Connell is more blessed than the rest, no? But yes, broadly speaking you are correct. While contemporary romantasy titles do seem to use Irish and Gaelic names and mythological creatures a fair bit, Ireland itself is not a mythical country. I have friends who claim to have been born there, and others who say they’ve been there. I’ve seen photos and everything.

However! I draw your attention to Connell’s career. While barely out of university, this working-class, unconnected youth wins literary prizes and is entertaining a career as a literary novelist. What is that if not fantasy?

So all the comments (correctly) went “well, that’s stupid. That book is not Romantasy. What are they even talking about?”

What they’re talking about is something that Norwegian has several words for, but that English has an odd lack of: Widely understood class based put downs for literature.

When I read Normal People some years back, my brain went “well, this is kiosklitteratur“. Which means, directly translated, “kiosk literature”. (Now you understand Norwegian!) I.e., a book that’s commonly sold in kiosks, bodegas and grocery stores. Not one of those Real Proper Books that’s sold in bookstores, right?

And reading oldee tymey literature critiques, English had a sorta equivalent term 80 years ago — “paperback literature” as a putdown. But as a term, it obviously had a time limit for being useful, because soon all books, even “proper” ones, were published in paperback, too. And there’s terms connected to various genres that can be used — “pulp novels” or “bodice rippers” — but since they’re tied to specific genres they aren’t useful as a general category.

Norwegians trying to be slightly less snooty about the category call it triviallitteratur, which means (yes, you guessed it “trivial literature”; you totally know Norwegian now), and that’s what I was thinking while reading the book: “Literary people are saying that this book is some kind of masterpiece — could it be that they’ve just never read triviallitteratur before and don’t know what it’s like?” Because I have, and the category has delights not found in “proper literature” that might very well appeal to literary people who don’t realise what they’re reading.

(“Romantasy” obviously belongs in the kiosklitteratur category, by the way.)

But for want of a term, that post about Normal People failed. (I agreed with everything it said otherwise, mind you.)

It’s so weird that English hasn’t grown terms in this area, because the UK certainly doesn’t lack class consciousness? But they also have to reach for French even to describe Booker prize candidates (“petit bourgeois literature”, and who can even remember how to spell booshwah?), and there’s no good word to express disdain for people who do things that Just Aren’t Done (harry in Norwegian). There’s the US “white trash”, but as many American things, it’s ickily racialised.

English is such an impoverished language, man.

Calendaring

The other week I made a calendrical display for the eink thing I have. The problem is that I don’t really use a calendar — I guess the main reason is because I hate planning, but the other reason is because I really dislike all the calendar entry interfaces I’ve used.

I haven’t used all that many, though, so I may well be missing some ingenious calendar out there. But what I’ve been resentfully using is the standard Google calendar on Android.

I filmed one of my interactions with this thing, and I swear — I didn’t try to make it more awkward than it is for me:

I’d be climbing up the walls if I were to try to use it more than I do now — I just put the absolute bare necessities in there, like dentist appointments and Boris shows. (One coming up in November!)

But now I have this calendar in the hall… wouldn’t it be nice to have it display more stuff? Stuff that I really should be remembering, anyway?

There’s also org-gcal, if you’re in the Org ecosystem… but… I’m not.

Does a command line interface to Google Calendar work? Because I want to have the calendar in my pocket, too. And… Yes, there’s gcalcli. (Which I noticed hasn’t been updated for two years and is probably going to be deprecated, but it works for now, so I haven’t bothered to redo my code.)

After going through the byzantine process on Google Cloud Console to create an “application”:

Yes! You have to create a logo! Behold my magnificent logo!

Anyway, after all that… tada:

Yes! The system works!

It does the normal Do What I Mean Thing — “fri 9” means “next Friday at 9am”, and “aug 11 1930” means “half past seven on August 11”, etc:

I’m freee! I never have to enter a calendar entry on the mobile phone ever again!

I’ve put the resulting code on Microsoft Github.