Spellchecking Total Eclipse with an AI

I’ve spent a day test-driving that Emacs minor mode for spellchecking using an LLM that I wrote the other day. I went through all the posts on the Total Eclipse blog (all 260 of them) to see whether the thing works or not.

And, yes, it works now, but it took some tweaking. Well, I rewrote the mode.

Twice.

My original approach was to give the LLM the entire text, and then ask it to return it to me verbatim, but with changed words tagged up specially. This worked 90% of the time, but it would randomly decide to do something else… and I was never totally sure that it actually returned the entire text. That approach was a bit too nerve-racking.

But Sacha Chua has done something along similar lines, and her approach is to ask the LLM to return just the changed parts:

That sounds like a much more responsible approach, so I redid my package to do pretty much the same… but no matter how I phrased things in the prompt, I couldn’t get the LLM to follow my instructions more than 50% of the time, really.

The hard thing to make it understand is that it should return a regexp that identifies the fragment uniquely, and that the word to be replaced should be inside a \(…\) group. But:

Some regexps would have zero grouping operators, and some would have several. And sometimes it would wrap the JSON in Markdown:

And worst of all, I could not get it to understand even once that it shouldn’t do this:

I.e., it seems like the LLM did the “regexps” by slapping a (…) around the term to be replaced, and then it put two backslashes before every parenthesis in the string. This is wrong.

So I redid the thing once again, but without regexps. Instead I asked it to return some JSON of the offending phrase (with enough context to uniquely identify it), and what it wants to replace it with:

And then I compute myself the actual difference between the phrases for markup purposes.

After this change, the LLM hasn’t screwed up the format once.

I guess what this really drives home for me is that the apparent enormous strides LLMs have taken the last six months for programming are mostly an illusion: You can now ask an LLM to whip you up, say, a Python script, and it chugs away at it, and then gives you a Python script that’s syntactically correct and mostly does what you asked.

But this isn’t because the LLMs now can generate correct syntax flawlessly. No, this is because the LLMs now have a sandbox they test-run all scripts before handing them over. So they discover that what they’ve generated is junk, and then iterate rapidly over it until it works.

But when I’m using the API here, it can’t check whether the regexps I asked it to generate are valid. It spits out something that’s close, but if it had its normal interactive harness, it would have tested the regexps and seen that they were invalid, and then fixed them.

So… it’s the LLM tooling that tricks us into believing that LLMs are pretty smart at computer stuff now. They aren’t, really.

Anyway, the Total Eclipse blog should now be 100% without typos! Woohoo! And since I’ve eyeballed every change, hopefully the LLM hasn’t changed too many instances of “important” to “load-bearing” without me noticing.

(One amusing thing I’ve noticed is that you can run the same article through the LLM several times, accepting the edits each time. As it finds fewer and fewer real things to complain about, it gets more and more nitpicky. But eventually it gives up and says “nothing to be changed”.)

Some LLM spell check experiences

A few days back, I hacked up a little minor mode for Emacs to use an LLM for spell checking. (Before you get a heart attack — I did not say “editing”, “getting ideas” or “research”: Just spell checking. Using an LLM for actual writing isn’t just a bad idea — it leads to abhorrent results.)

I’ve been running a random selection of blog posts I wrote for the Comico read through blog to see whether the thing works well — and… yes! But also no.

OK, positives first:

1) For simple typos, it’s very good indeed — it finds them all.

c) For word substitution errors, like “to” instead of “too”, it’s also very good.

II) For grammar errors, it’s more mixed. It’s good at, for instance, basic noun/verb agreement, so if I’ve written “they writes”, it fixes that. But if the noun clause is long and complex, it doesn’t always react. I’ve experimented with adding/removing an “s” from the verb in sentences like that, and it sometimes lets both versions through without comment.

⁵) Since I’m usually giving the LLM the complete blog post to proof-read, it knows what area I’m writing about, and so it’s able to correct me when I misspell names, too. I hadn’t even considered that, but that was a very nice surprise. So when I’m writing about comics and casually mention “the art looks a bit like Kriegstein”, it knows to correct that to “Krigstein”. This is something that traditional spell check can’t do at all. (And amusingly enough, running this article through the LLM doesn’t tag that “they writes” up there as an error, because it gets that that was an example of an error.)

So that’s all very good. So what’s the bad bits? Well, it’s an LLM, so it’s slow. That’s not a major problem, though, since I’m just using it to give a blog post a look-over to weed out the worst spelling mistakes. But I’ve added some code to strip out (longer) HTML fragments before sending the text over to the LLM, and this helps a bit.

The biggest problem is that an LLM is an LLM and will randomly decide to not follow the instructions in the prompt. Sometimes it adds random elements to the text here and there. Sometimes it says that I should fix a word from “identical” to “identical”. Sometimes it removes a paragraph. This doesn’t happen often — perhaps in 5% of the cases? But…

LLMs are unreliable! It cannot be! *gasp*

(Somebody speculated that this unreliability is a feature for the LLM companies — because you know that you may get bad results, you get extra happy the times you get a good result. The slot machine thing? I think that’s probably the case — some people get really addicted to their LLMs…)

Fortunately, these things can be programmed around — I’ve now added code to do sanity checks to the output from the LLM before using it. So if the LLM decides to go off piste, the package detects that, and then you can just do the query again.

It used to be that doing the same thing and expecting a different result was a sign of insanity, but with an LLM, that’s fine.

It’s a brave new world out there.

It’s also … odd … to type commands in Emacs and then they cost money.

Anyway, I’ve pushed these fixes to Microsoft Github, so there you go.

English is such an impoverished language

That’s what I say, jokingly, every time I discover that English doesn’t have that particular word I’m looking for to distinguish between two natural concepts.

And it’s a joke, because of course English is a huge language, what with it swallowing up several other languages wholesale and then just misspelling all the words a lot. (It’s something Norwegians seriously say about Norwegian, though.)

But, like, for instance — many languages have words to distinguish between “a 24 hour period” and “a day”, and English doesn’t, so that’s awkward. And Norwegian has different words for “acting suspicious” and “being suspicious of others”.

English is such an impoverished language, man!

(Here’s where you pull out your OED and say “but see? here’s one!” and point to a word that has three hits on Google as evidence that English really does have that word.)

Joking aside, I there’s one area where English really does seem to have a lack of words/concepts that do make a difference — where you can see people are trying to say something about an issue, but the words to describe the issue just aren’t there. Not to go all Whorfian on you, but the other week I read an article that just made me go *GNGNGNG* because I was agreeing with everything it was trying to say, but it seemed to lack the concepts of what it was describing.

It was this article about Sally Rooney’s novel Normal People, and it describes the book as “romantasy”:

‘Right,’ you say. ‘So that’s surely not Normal People, which is set in 2010s Ireland. Not a dragon or a well-endowed faerie prince in sight.’

Which, I mean. I don’t know about that last part. Normal People certainly gives us to suspect Connell is more blessed than the rest, no? But yes, broadly speaking you are correct. While contemporary romantasy titles do seem to use Irish and Gaelic names and mythological creatures a fair bit, Ireland itself is not a mythical country. I have friends who claim to have been born there, and others who say they’ve been there. I’ve seen photos and everything.

However! I draw your attention to Connell’s career. While barely out of university, this working-class, unconnected youth wins literary prizes and is entertaining a career as a literary novelist. What is that if not fantasy?

So all the comments (correctly) went “well, that’s stupid. That book is not Romantasy. What are they even talking about?”

What they’re talking about is something that Norwegian has several words for, but that English has an odd lack of: Widely understood class based put downs for literature.

When I read Normal People some years back, my brain went “well, this is kiosklitteratur“. Which means, directly translated, “kiosk literature”. (Now you understand Norwegian!) I.e., a book that’s commonly sold in kiosks, bodegas and grocery stores. Not one of those Real Proper Books that’s sold in bookstores, right?

And reading oldee tymey literature critiques, English had a sorta equivalent term 80 years ago — “paperback literature” as a putdown. But as a term, it obviously had a time limit for being useful, because soon all books, even “proper” ones, were published in paperback, too. And there’s terms connected to various genres that can be used — “pulp novels” or “bodice rippers” — but since they’re tied to specific genres they aren’t useful as a general category.

Norwegians trying to be slightly less snooty about the category call it triviallitteratur, which means (yes, you guessed it “trivial literature”; you totally know Norwegian now), and that’s what I was thinking while reading the book: “Literary people are saying that this book is some kind of masterpiece — could it be that they’ve just never read triviallitteratur before and don’t know what it’s like?” Because I have, and the category has delights not found in “proper literature” that might very well appeal to literary people who don’t realise what they’re reading.

(“Romantasy” obviously belongs in the kiosklitteratur category, by the way.)

But for want of a term, that post about Normal People failed. (I agreed with everything it said otherwise, mind you.)

It’s so weird that English hasn’t grown terms in this area, because the UK certainly doesn’t lack class consciousness? But they also have to reach for French even to describe Booker prize candidates (“petit bourgeois literature”, and who can even remember how to spell booshwah?), and there’s no good word to express disdain for people who do things that Just Aren’t Done (harry in Norwegian). There’s the US “white trash”, but as many American things, it’s ickily racialised.

English is such an impoverished language, man.

Calendaring

The other week I made a calendrical display for the eink thing I have. The problem is that I don’t really use a calendar — I guess the main reason is because I hate planning, but the other reason is because I really dislike all the calendar entry interfaces I’ve used.

I haven’t used all that many, though, so I may well be missing some ingenious calendar out there. But what I’ve been resentfully using is the standard Google calendar on Android.

I filmed one of my interactions with this thing, and I swear — I didn’t try to make it more awkward than it is for me:

I’d be climbing up the walls if I were to try to use it more than I do now — I just put the absolute bare necessities in there, like dentist appointments and Boris shows. (One coming up in November!)

But now I have this calendar in the hall… wouldn’t it be nice to have it display more stuff? Stuff that I really should be remembering, anyway?

There’s also org-gcal, if you’re in the Org ecosystem… but… I’m not.

Does a command line interface to Google Calendar work? Because I want to have the calendar in my pocket, too. And… Yes, there’s gcalcli. (Which I noticed hasn’t been updated for two years and is probably going to be deprecated, but it works for now, so I haven’t bothered to redo my code.)

After going through the byzantine process on Google Cloud Console to create an “application”:

Yes! You have to create a logo! Behold my magnificent logo!

Anyway, after all that… tada:

Yes! The system works!

It does the normal Do What I Mean Thing — “fri 9” means “next Friday at 9am”, and “aug 11 1930” means “half past seven on August 11”, etc:

I’m freee! I never have to enter a calendar entry on the mobile phone ever again!

I’ve put the resulting code on Microsoft Github.

Last Week’s Comics Tonight

Here’s some comics I’ve read over the past week.

I’ve enjoyed several comics written by Matt Fraction, so I gave this a go — even though I’ve barely read any Iron Man comics since I was… hm… I may actually never have been reading Iron Man comics ever? I mean, I read Avengers and stuff, but never the Iron Man title.

But… like… is that what Stark looks like now? Or, I mean, 15 years ago? It’s just odd.

And the book is pretty dull. The entire thing has one super duper villain that’s plotting against Iron Man, but it’s so… stupid? And then Fraction spends a lot of time on fake drama like the above, possibly to divert attention from “The Mandarin” being a horribly boring villain.

I was wondering whether Fraction was aiming the book towards, like, twelve-year-olds — if he were, I would be applauding the effort. But it doesn’t really seem like he is? Instead the book is just kinda stupid?

But back to Larroca’s artwork — the most fascinating thing about the book is the way Stark’s nose changes size all the time. I mean, it’s generally pretty big, but sometimes it’s 2x the size. Which is something you see often when artists are snapping reference photos too close up, which again happens when they use themselves as the model. (Too short arms.) So I didn’t google this, but now I’m gonna! What does Larroca look like!

Hah! I’m a genius comics art sleuth!

This is a brick of a book — over 400 pages. It’s a German comic made by Nando von Arb and originally called Fürchten Lernen. You don’t see many German comics…

And I really enjoyed the artwork.

It has an impact and it’s got something going on.

The book is about various mental health issues, and while it’s a good read, it does go over some extremely well-trodden ground for comics.

When I heard that Jim Woodring had a new book out, I was excited, but I’ve registered zero chatter about Quacky on the intertubes.

And it’s not in Woodring’s normal style (which tends towards wordlessness). Instead it’s done in the Little Big Books style, I guess? I’ve never seen one myself, but…

So there’s a text on the verso page and an illustration on the recto page. And then a sentence beneath that that recaps it all — sometimes in a slightly mischievous way.

At first I went “noooo”, because I’ve seldom seen a successful book done in this way… but then I got really into it. You work up a rhythm, and it all becomes rather intriguing.

The Frank stories usually are pretty distressing, but this one is less distressing than most. And then there’s a shorter story in the back which is much weirder.

So I liked it a lot. And of course Woodring’s artwork is fantastic as usual.

The only thing I wonder about — what happened to the expensive plant! Did Frank take care of it!? THE PUBLIC DESERVES TO KNOW

This book by Rotem Anna Diamant is something else again.

I like the general approach of the book…

… but the fairy tale/myth thing has never been my thing, so I didn’t really take to this book, even though I like the art.

A Swedish book — Taggig by Alva Skog.

And yes, as you already guessed from the cover, there’s a lotta gender goin’ on, as they say.

I really enjoyed this book. It’s got such an easy flow — it doesn’t get bogged down, but instead just meanders easily, taking in the story points. Really good storytelling chops.

And attractive artwork.

I’ve read some Achille Talon in translation as a child, but I didn’t quite remember what it was like. But I found this in the used comics store, so I picked it up.

And… yes, the gag here is that Talon (and most of the other characters) never stop spouting verbiage. So it’s one over-the-top dialogue after another, and most of it goes over my head — I can barely make sense of the surface level dialogue, never mind make out all the jokes. Reading a comic that mangles the language in this way isn’t ideal for a French beginner like me.

But it is very funny. Funnier than I think I remember it being when I read it as a child. (Although I never read this album, I think.) Took me several hours to get through it, though.

I’ve not heard of Berthet or Tome before…

… but this album is from the early 90s, and it’s about a favourite subject of French comics — racism, sex and violence in the Deep South of the US.

It’s not bad… and it avoids some of the clichés. But it’s not… very interesting.

And! I read a bunch of super-hero books.

I started a Flash subscription because Ryan North started writing it, and that was a good decision, because it’s pretty entertaining. But it’s a very Ryan book — that is, The Flash gets lots of incredible new powers and stuff, and while that’s fun, you have to wonder about the long term viability of this stuff.

Like, his Fantastic Four are now so powerful they can pretty much do anything… so the stories veer between being way too Everything Is At Stake Now and Let’s Just Have Fun in a rather bewildering fashion.

But it is still fun.

The pretty bad Planet She-Hulk series ends in a bizarre way — with She-Hulk being back on Earth, and having total amnesia about it all. But somebody from that planet drops by and recaps what (I’m assuming) were to have happened in the series. I’m guessing this is because the series was cancelled? So it’s nice of them to give us a “this is what was going to happen” issue… and it kinda turned out to be the best issue?

OK, that’s it.